COMPREHENSIVE EXAM QUESTIONS IN RESEARCH METHODS(1)

1 PHILOSOPHY LOGIC AND ETHICS OF SCIENCE

A) General

A1 Starbuck has suggested that the properties shared by all organizations ought to be uninteresting

and unimportant He also notes that although statements about averages bother very few they

ought to bother many (Journal of Management Studies vol 30(6) 1993)

Do you agree with these statements If so why and what are the implications of your beliefs for

research methods in strategic management and organization science If not why not and what

are the implications of your beliefs for research methods in those areas

TH

A2 Lay out the components of the Runkel and McGrath (1972) Research Cycle and the Martin

(1982) Garbage Can Model of the research process What are the aims of either approach What

assumptions does each approach make What basic themes does each approach emphasize

What kinds of constraints on the process of doing research does each approach highlight How

are the two approaches alike and different in other ways What are the likely pitfalls of viewing

the research process from only one of these two viewpoints

A3 Define (and discuss important features or issues concerning) the following constructs of

Philosophy and Logic of Science (Define any four terms)

1 Null hypothesis 6 Nomological network

2 Logical empiricism (positivism) 7 Modus tollens

3 Paradigm 8 The fallacy of affirming the consequent

4 Independent variable 9 Theory

5 Operational definition

A4 Organizational Studies may be described as either a basic or an applied

discipline From your point of view is Organizational Studies a basic discipline

or an applied discipline or both What should it be When answering this

question be sure to define distinctions between these two types of disciplines

Provide evidence from the organizational studies literature that supports your

position

A5 What rules of evidence are used to determine whether a research finding is

meaningful

A6 Considerable debate in the social and behavioral sciences has occurred

concerning the role of the context of discovery versus the context of

justification Much of the debate has addressed the following sets of issues

a What features distinguish the role of discovery from the role of justification

b What is the role of discovery versus justification in the research process

c Compare the perspectives of a logical empiricist and a relativist What

distinctions would they make between the context of discovery and the context

of justification

d Can a discipline be said to progress within the context of discovery

A7 An epistemology is a set of rules for the appropriate way to generate knowledge

Outline the major components of your epistemology for Organizational Studies

research You may wish to proceed by identifying one of the established

epistemologies and comparing it to yours (eg skepticism positivism

empiricism pragmatism relativism) Describe the process by which you feel

knowledge can be generated most effectively in Organizational Studies Identify

and define the key concepts of your research philosophy and discuss its

strengths and weaknesses

A8 Abraham Kaplan argues that behavioral scientists should practice openness in

inquiry that is they should not pre-judge research or theory negatively or

positively on the basis of its content There are many other competing

philosophical schools of thought as to how empirical research should be

conducted Much of the debate has addressed the following issues

a What are the criteria for judging whether research is worthwhile Is there

some research that should not even be done

b What is the most effective model of the research process

c How does a researcher determine the meaningfulness of a research finding

That is what are the rational rules of evidence necessary for evaluating research

findings

In your doctoral studies you have been exposed to a variety of approaches for

addressing each of the above issues Please describe in detail your position on

these issues and provide citations where appropriate In addition please

highlight the major competing positions on each issue

A9 In recent years there has been considerable debate in the social and behavioral

sciences concerning the two competing epistemologies-- logical empiricism

(including falsificationism) and relativism (ie post-positivism)

a What characteristics distinguish these two epistemologies Be sure to address

both conceptual and methodological differences

b Take the perspective of a logical empiricist How would this perspective

affect the manner in which you conduct research Be sure to discuss at least the

following issues a) what model(s) would you use to approach the research

process b) theory evaluation and c) data collection

c Now take the perspective of a relativist Address the same issues raised in part

b

A10 In recent years there has been considerable debate in the social and behavioral

sciences concerning two competing research paradigms--one based on a logical

empiricismrealist epistemology and the second based on a post-empiricist

epistemology These paradigms vary with respect to their underlying aims

(goals) standards for judging explanations and ontology

a Describe the basic aims standards and ontology for each paradigm What

characteristics distinguish these two paradigms

b One current concern focuses on the appropriateness of a priori and post-hoc

explanations for theory testing Describe and defend your point-of-view

concerning which type of explanation is appropriate for research in

organizational studies

c A second current concern focuses on the appropriate treatment of the

independent variables that is an experimental approach (where the independent

variables are manipulated) and an observational approach (where the

independent variables are observed) Describe and defend your point-of-view

concerning which treatment of the independent variables is likely to produce the

more important contribution to organizational studies

A11 What rules of evidence are used by the positivist research paradigm to determine

whether a research finding is meaningful Choose another research

philosophyparadigm and compare and contrast it to the positivist rules of

evidence

A12 A major tenet of research is that research subjects should have limited if any

knowledge of the subject of the research The assumption is that subjects will

not act naturally if they are aware of the purpose of the research For this reason

researchers often engage in various types of deception For example

experimental and survey researchers will word instructions in such a way as to

hide the research goals and researchers who conduct participant observation

will avoid telling people they are researchers Discuss the ethical questions

raised with deception What guidelines would you give to a colleague who was

conducting research that involved deception

B) Causation

1) Explain the phrase ldquocorrelation does not equal causationrdquo Why is the distinction so

important What conditions must be met to establish causation Can they ever be

completely satisfied

12) Why is a posttest only design without random assignment to treatment conditions

uninterpretable from a causal standpoint

B1 Choose a model from the Organizational Behavior literature that has causal

orderings between variables and has been tested in a field setting Compare the

following data analytic methods for testing hypotheses about causality between

variable sets (eg dependent independent intervening moderating exogenous

endogenous)

a) path analysis using hierarchical regression

b) maximum likelihood estimates of structural equations models (perhaps through

LISREL)

Your answer should address

1 The parameters the procedures estimate and how they are estimated

2 How the validity of the models are judged

3 Appropriate inferences of causality and conditions necessary to make such

inferences

4 The statistical assumptions underlying each technique and the consequences of

violating assumptions

B2 Describe the critical-realist perspective of Cook and Campbell (1979) Compare and

contrast this perspective on causality with that of Poppers falsificationism

B3 According to Cook and Campbell (1979) causal relationships in the social sciences

can be probabilistic and still meet the assumption of causality that asserts that a

cause must always produce an effect Describe the logic they invoke to justify this

position

B4 Identify James Mulaik amp Bretts (1982) conditions for establishing causal

inference Discuss what they mean and why they are important to causal inference

B5 What are the three conditions necessary to establish causality Can these conditions

ever be fully met Why or why not

B6 Assume you are interested in conducting a study that allows you to legitimately

make some causal inferences concerning relationships among a set of constructs

What sorts of things can you as the researcher do to bolster your ability to make

such causal inferences In answering this question be sure to pay attention to issues

pertaining to both study design and data analysis

B7 Some researchers (and editorsreviewers) are very critical about the ease with which

others use causal language in interpreting their analyses What are the concerns

behind this criticism and what are the methodological and statistical conditions that

give rise to these concerns Under what conditions can researcher make reasonable

valid causal interpretations of their findings Can a researcher ever be completely

confident making such causal statements Be sure to defend your position in

answering these questions

C) Theory amp Research

C1 What are the elements of a theory What criteria did Bacharach (1989) use to

evaluate if a theory is good

IC

C2 The Academy of Management Journal in its Information for Contributors

states All articles published in the Journal must make a strong theoretical

contribution Because this is one of our top publications we must have a clear

understanding of what constitutes a strong theory Describe the essential elements

of a good theory You can also describe practices that are not indicative of good

theory if that will help you articulate what is good theory

C3 Organizational researchers place a high premium on research that is guided by

theory This perspective is evident in journals such as AMJ and ASQ which will

often reject a manuscript that describes a sound empirical study but does not

provide a theoretical contribution Is there a role for atheoretical research in the

organizational sciences If so what is the role of atheoretical research and what

contribution does it make to knowledge in the organizational sciences

TH

C4 Define inductive and deductive methods for generating knowledge What are the

advantages and disadvantages of each Under what conditions is each

appropriate

C5 Daft (1985) in Why I Recommended That Your Manuscript Be Rejected and

What You Can Do About It said that poor theoretical development is the

primary reason he recommended manuscripts be rejected when reviewing for

AMJ and ASQ

Why is theory considered so important

Give an example of a good theory and explain the elements that make it a strong

theory (Focus on theory development not on the content of the specific theory)

C6 You have undoubtedly heard the criticism of certain published articles in our

field being atheoretical Spoken as a criticism it suggests that research articles

should have theory Should every empirical article have theory Why is theory

important What are the hallmarks of good theory

2 RESEARCH DESIGN

A) Research Design Terms

5) Define internal and external validity Which is more important and why (It is not okay to

say they are equally important)

A1 Define (and discuss important features or issues concerning) the following

constructs of research design (Define any five terms)

1 Internal validity 6 Quasi-experimental

design

2 Stratified Sampling 7 Random Assignment

3 Solomon Four-group

design 8 Differential mortality

4 Factorial design 9 Demand

characteristic

5 External validity 10 Regression to the

mean

B) Level of Analysis

B1 IO psychologists have traditionally categorized themselves as being more micro or

macro in orientation usually falling toward the micro end Recent concern for what

has been termed the levels of analysis or cross-level approach has added a third

category - meso What is the meso level Is this just academic hairsplitting or re-

naming or is there a fundamental difference in this research strategy in contrast to

micro or macro Provide an example from some research domain of interest to you

that illustrates your conclusion of no real difference or important difference

C) Cross-Sectional vs Longitudinal Design

13) What are the benefits of a longitudinal design over a cross-sectional design in a non

experimental (correlational) study What are the disadvantages Why donrsquot we see

more longitudinal research

C1 The use of cross-sectional research designs continues despite many calls for

replacing them with longitudinal designs (a) What are the main strengths of

cross-sectional designs (b) What weaknesses do they have that longitudinal

designs supposedly counter (c) What are the implications of cross-sectional

designs for theory development (Consider internal and external validity)

C2 A good number of research efforts on the micro (OBHR) side rely on cross-

sectional designs in which surveys are used to collect both IVs and DVs from the

same respondent What are the various problems associated with such a design

What changes could be made to this design to alleviate andor eliminate these

problems What are the downsides of the new design (add in conveniencenon

probability samples)

C3 What are some of the major strengths of a longitudinal design Why dont we see

more of them

C4 Assume that you are analyzing a large sample of longitudinal survey data TH

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

A10 In recent years there has been considerable debate in the social and behavioral

sciences concerning two competing research paradigms--one based on a logical

empiricismrealist epistemology and the second based on a post-empiricist

epistemology These paradigms vary with respect to their underlying aims

(goals) standards for judging explanations and ontology

a Describe the basic aims standards and ontology for each paradigm What

characteristics distinguish these two paradigms

b One current concern focuses on the appropriateness of a priori and post-hoc

explanations for theory testing Describe and defend your point-of-view

concerning which type of explanation is appropriate for research in

organizational studies

c A second current concern focuses on the appropriate treatment of the

independent variables that is an experimental approach (where the independent

variables are manipulated) and an observational approach (where the

independent variables are observed) Describe and defend your point-of-view

concerning which treatment of the independent variables is likely to produce the

more important contribution to organizational studies

A11 What rules of evidence are used by the positivist research paradigm to determine

whether a research finding is meaningful Choose another research

philosophyparadigm and compare and contrast it to the positivist rules of

evidence

A12 A major tenet of research is that research subjects should have limited if any

knowledge of the subject of the research The assumption is that subjects will

not act naturally if they are aware of the purpose of the research For this reason

researchers often engage in various types of deception For example

experimental and survey researchers will word instructions in such a way as to

hide the research goals and researchers who conduct participant observation

will avoid telling people they are researchers Discuss the ethical questions

raised with deception What guidelines would you give to a colleague who was

conducting research that involved deception

B) Causation

1) Explain the phrase ldquocorrelation does not equal causationrdquo Why is the distinction so

important What conditions must be met to establish causation Can they ever be

completely satisfied

12) Why is a posttest only design without random assignment to treatment conditions

uninterpretable from a causal standpoint

B1 Choose a model from the Organizational Behavior literature that has causal

orderings between variables and has been tested in a field setting Compare the

following data analytic methods for testing hypotheses about causality between

variable sets (eg dependent independent intervening moderating exogenous

endogenous)

a) path analysis using hierarchical regression

b) maximum likelihood estimates of structural equations models (perhaps through

LISREL)

Your answer should address

1 The parameters the procedures estimate and how they are estimated

2 How the validity of the models are judged

3 Appropriate inferences of causality and conditions necessary to make such

inferences

4 The statistical assumptions underlying each technique and the consequences of

violating assumptions

B2 Describe the critical-realist perspective of Cook and Campbell (1979) Compare and

contrast this perspective on causality with that of Poppers falsificationism

B3 According to Cook and Campbell (1979) causal relationships in the social sciences

can be probabilistic and still meet the assumption of causality that asserts that a

cause must always produce an effect Describe the logic they invoke to justify this

position

B4 Identify James Mulaik amp Bretts (1982) conditions for establishing causal

inference Discuss what they mean and why they are important to causal inference

B5 What are the three conditions necessary to establish causality Can these conditions

ever be fully met Why or why not

B6 Assume you are interested in conducting a study that allows you to legitimately

make some causal inferences concerning relationships among a set of constructs

What sorts of things can you as the researcher do to bolster your ability to make

such causal inferences In answering this question be sure to pay attention to issues

pertaining to both study design and data analysis

B7 Some researchers (and editorsreviewers) are very critical about the ease with which

others use causal language in interpreting their analyses What are the concerns

behind this criticism and what are the methodological and statistical conditions that

give rise to these concerns Under what conditions can researcher make reasonable

valid causal interpretations of their findings Can a researcher ever be completely

confident making such causal statements Be sure to defend your position in

answering these questions

C) Theory amp Research

C1 What are the elements of a theory What criteria did Bacharach (1989) use to

evaluate if a theory is good

IC

C2 The Academy of Management Journal in its Information for Contributors

states All articles published in the Journal must make a strong theoretical

contribution Because this is one of our top publications we must have a clear

understanding of what constitutes a strong theory Describe the essential elements

of a good theory You can also describe practices that are not indicative of good

theory if that will help you articulate what is good theory

C3 Organizational researchers place a high premium on research that is guided by

theory This perspective is evident in journals such as AMJ and ASQ which will

often reject a manuscript that describes a sound empirical study but does not

provide a theoretical contribution Is there a role for atheoretical research in the

organizational sciences If so what is the role of atheoretical research and what

contribution does it make to knowledge in the organizational sciences

TH

C4 Define inductive and deductive methods for generating knowledge What are the

advantages and disadvantages of each Under what conditions is each

appropriate

C5 Daft (1985) in Why I Recommended That Your Manuscript Be Rejected and

What You Can Do About It said that poor theoretical development is the

primary reason he recommended manuscripts be rejected when reviewing for

AMJ and ASQ

Why is theory considered so important

Give an example of a good theory and explain the elements that make it a strong

theory (Focus on theory development not on the content of the specific theory)

C6 You have undoubtedly heard the criticism of certain published articles in our

field being atheoretical Spoken as a criticism it suggests that research articles

should have theory Should every empirical article have theory Why is theory

important What are the hallmarks of good theory

2 RESEARCH DESIGN

A) Research Design Terms

5) Define internal and external validity Which is more important and why (It is not okay to

say they are equally important)

A1 Define (and discuss important features or issues concerning) the following

constructs of research design (Define any five terms)

1 Internal validity 6 Quasi-experimental

design

2 Stratified Sampling 7 Random Assignment

3 Solomon Four-group

design 8 Differential mortality

4 Factorial design 9 Demand

characteristic

5 External validity 10 Regression to the

mean

B) Level of Analysis

B1 IO psychologists have traditionally categorized themselves as being more micro or

macro in orientation usually falling toward the micro end Recent concern for what

has been termed the levels of analysis or cross-level approach has added a third

category - meso What is the meso level Is this just academic hairsplitting or re-

naming or is there a fundamental difference in this research strategy in contrast to

micro or macro Provide an example from some research domain of interest to you

that illustrates your conclusion of no real difference or important difference

C) Cross-Sectional vs Longitudinal Design

13) What are the benefits of a longitudinal design over a cross-sectional design in a non

experimental (correlational) study What are the disadvantages Why donrsquot we see

more longitudinal research

C1 The use of cross-sectional research designs continues despite many calls for

replacing them with longitudinal designs (a) What are the main strengths of

cross-sectional designs (b) What weaknesses do they have that longitudinal

designs supposedly counter (c) What are the implications of cross-sectional

designs for theory development (Consider internal and external validity)

C2 A good number of research efforts on the micro (OBHR) side rely on cross-

sectional designs in which surveys are used to collect both IVs and DVs from the

same respondent What are the various problems associated with such a design

What changes could be made to this design to alleviate andor eliminate these

problems What are the downsides of the new design (add in conveniencenon

probability samples)

C3 What are some of the major strengths of a longitudinal design Why dont we see

more of them

C4 Assume that you are analyzing a large sample of longitudinal survey data TH

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

B1 Choose a model from the Organizational Behavior literature that has causal

orderings between variables and has been tested in a field setting Compare the

following data analytic methods for testing hypotheses about causality between

variable sets (eg dependent independent intervening moderating exogenous

endogenous)

a) path analysis using hierarchical regression

b) maximum likelihood estimates of structural equations models (perhaps through

LISREL)

Your answer should address

1 The parameters the procedures estimate and how they are estimated

2 How the validity of the models are judged

3 Appropriate inferences of causality and conditions necessary to make such

inferences

4 The statistical assumptions underlying each technique and the consequences of

violating assumptions

B2 Describe the critical-realist perspective of Cook and Campbell (1979) Compare and

contrast this perspective on causality with that of Poppers falsificationism

B3 According to Cook and Campbell (1979) causal relationships in the social sciences

can be probabilistic and still meet the assumption of causality that asserts that a

cause must always produce an effect Describe the logic they invoke to justify this

position

B4 Identify James Mulaik amp Bretts (1982) conditions for establishing causal

inference Discuss what they mean and why they are important to causal inference

B5 What are the three conditions necessary to establish causality Can these conditions

ever be fully met Why or why not

B6 Assume you are interested in conducting a study that allows you to legitimately

make some causal inferences concerning relationships among a set of constructs

What sorts of things can you as the researcher do to bolster your ability to make

such causal inferences In answering this question be sure to pay attention to issues

pertaining to both study design and data analysis

B7 Some researchers (and editorsreviewers) are very critical about the ease with which

others use causal language in interpreting their analyses What are the concerns

behind this criticism and what are the methodological and statistical conditions that

give rise to these concerns Under what conditions can researcher make reasonable

valid causal interpretations of their findings Can a researcher ever be completely

confident making such causal statements Be sure to defend your position in

answering these questions

C) Theory amp Research

C1 What are the elements of a theory What criteria did Bacharach (1989) use to

evaluate if a theory is good

IC

C2 The Academy of Management Journal in its Information for Contributors

states All articles published in the Journal must make a strong theoretical

contribution Because this is one of our top publications we must have a clear

understanding of what constitutes a strong theory Describe the essential elements

of a good theory You can also describe practices that are not indicative of good

theory if that will help you articulate what is good theory

C3 Organizational researchers place a high premium on research that is guided by

theory This perspective is evident in journals such as AMJ and ASQ which will

often reject a manuscript that describes a sound empirical study but does not

provide a theoretical contribution Is there a role for atheoretical research in the

organizational sciences If so what is the role of atheoretical research and what

contribution does it make to knowledge in the organizational sciences

TH

C4 Define inductive and deductive methods for generating knowledge What are the

advantages and disadvantages of each Under what conditions is each

appropriate

C5 Daft (1985) in Why I Recommended That Your Manuscript Be Rejected and

What You Can Do About It said that poor theoretical development is the

primary reason he recommended manuscripts be rejected when reviewing for

AMJ and ASQ

Why is theory considered so important

Give an example of a good theory and explain the elements that make it a strong

theory (Focus on theory development not on the content of the specific theory)

C6 You have undoubtedly heard the criticism of certain published articles in our

field being atheoretical Spoken as a criticism it suggests that research articles

should have theory Should every empirical article have theory Why is theory

important What are the hallmarks of good theory

2 RESEARCH DESIGN

A) Research Design Terms

5) Define internal and external validity Which is more important and why (It is not okay to

say they are equally important)

A1 Define (and discuss important features or issues concerning) the following

constructs of research design (Define any five terms)

1 Internal validity 6 Quasi-experimental

design

2 Stratified Sampling 7 Random Assignment

3 Solomon Four-group

design 8 Differential mortality

4 Factorial design 9 Demand

characteristic

5 External validity 10 Regression to the

mean

B) Level of Analysis

B1 IO psychologists have traditionally categorized themselves as being more micro or

macro in orientation usually falling toward the micro end Recent concern for what

has been termed the levels of analysis or cross-level approach has added a third

category - meso What is the meso level Is this just academic hairsplitting or re-

naming or is there a fundamental difference in this research strategy in contrast to

micro or macro Provide an example from some research domain of interest to you

that illustrates your conclusion of no real difference or important difference

C) Cross-Sectional vs Longitudinal Design

13) What are the benefits of a longitudinal design over a cross-sectional design in a non

experimental (correlational) study What are the disadvantages Why donrsquot we see

more longitudinal research

C1 The use of cross-sectional research designs continues despite many calls for

replacing them with longitudinal designs (a) What are the main strengths of

cross-sectional designs (b) What weaknesses do they have that longitudinal

designs supposedly counter (c) What are the implications of cross-sectional

designs for theory development (Consider internal and external validity)

C2 A good number of research efforts on the micro (OBHR) side rely on cross-

sectional designs in which surveys are used to collect both IVs and DVs from the

same respondent What are the various problems associated with such a design

What changes could be made to this design to alleviate andor eliminate these

problems What are the downsides of the new design (add in conveniencenon

probability samples)

C3 What are some of the major strengths of a longitudinal design Why dont we see

more of them

C4 Assume that you are analyzing a large sample of longitudinal survey data TH

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

C) Theory amp Research

C1 What are the elements of a theory What criteria did Bacharach (1989) use to

evaluate if a theory is good

IC

C2 The Academy of Management Journal in its Information for Contributors

states All articles published in the Journal must make a strong theoretical

contribution Because this is one of our top publications we must have a clear

understanding of what constitutes a strong theory Describe the essential elements

of a good theory You can also describe practices that are not indicative of good

theory if that will help you articulate what is good theory

C3 Organizational researchers place a high premium on research that is guided by

theory This perspective is evident in journals such as AMJ and ASQ which will

often reject a manuscript that describes a sound empirical study but does not

provide a theoretical contribution Is there a role for atheoretical research in the

organizational sciences If so what is the role of atheoretical research and what

contribution does it make to knowledge in the organizational sciences

TH

C4 Define inductive and deductive methods for generating knowledge What are the

advantages and disadvantages of each Under what conditions is each

appropriate

C5 Daft (1985) in Why I Recommended That Your Manuscript Be Rejected and

What You Can Do About It said that poor theoretical development is the

primary reason he recommended manuscripts be rejected when reviewing for

AMJ and ASQ

Why is theory considered so important

Give an example of a good theory and explain the elements that make it a strong

theory (Focus on theory development not on the content of the specific theory)

C6 You have undoubtedly heard the criticism of certain published articles in our

field being atheoretical Spoken as a criticism it suggests that research articles

should have theory Should every empirical article have theory Why is theory

important What are the hallmarks of good theory

2 RESEARCH DESIGN

A) Research Design Terms

5) Define internal and external validity Which is more important and why (It is not okay to

say they are equally important)

A1 Define (and discuss important features or issues concerning) the following

constructs of research design (Define any five terms)

1 Internal validity 6 Quasi-experimental

design

2 Stratified Sampling 7 Random Assignment

3 Solomon Four-group

design 8 Differential mortality

4 Factorial design 9 Demand

characteristic

5 External validity 10 Regression to the

mean

B) Level of Analysis

B1 IO psychologists have traditionally categorized themselves as being more micro or

macro in orientation usually falling toward the micro end Recent concern for what

has been termed the levels of analysis or cross-level approach has added a third

category - meso What is the meso level Is this just academic hairsplitting or re-

naming or is there a fundamental difference in this research strategy in contrast to

micro or macro Provide an example from some research domain of interest to you

that illustrates your conclusion of no real difference or important difference

C) Cross-Sectional vs Longitudinal Design

13) What are the benefits of a longitudinal design over a cross-sectional design in a non

experimental (correlational) study What are the disadvantages Why donrsquot we see

more longitudinal research

C1 The use of cross-sectional research designs continues despite many calls for

replacing them with longitudinal designs (a) What are the main strengths of

cross-sectional designs (b) What weaknesses do they have that longitudinal

designs supposedly counter (c) What are the implications of cross-sectional

designs for theory development (Consider internal and external validity)

C2 A good number of research efforts on the micro (OBHR) side rely on cross-

sectional designs in which surveys are used to collect both IVs and DVs from the

same respondent What are the various problems associated with such a design

What changes could be made to this design to alleviate andor eliminate these

problems What are the downsides of the new design (add in conveniencenon

probability samples)

C3 What are some of the major strengths of a longitudinal design Why dont we see

more of them

C4 Assume that you are analyzing a large sample of longitudinal survey data TH

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

A1 Define (and discuss important features or issues concerning) the following

constructs of research design (Define any five terms)

1 Internal validity 6 Quasi-experimental

design

2 Stratified Sampling 7 Random Assignment

3 Solomon Four-group

design 8 Differential mortality

4 Factorial design 9 Demand

characteristic

5 External validity 10 Regression to the

mean

B) Level of Analysis

B1 IO psychologists have traditionally categorized themselves as being more micro or

macro in orientation usually falling toward the micro end Recent concern for what

has been termed the levels of analysis or cross-level approach has added a third

category - meso What is the meso level Is this just academic hairsplitting or re-

naming or is there a fundamental difference in this research strategy in contrast to

micro or macro Provide an example from some research domain of interest to you

that illustrates your conclusion of no real difference or important difference

C) Cross-Sectional vs Longitudinal Design

13) What are the benefits of a longitudinal design over a cross-sectional design in a non

experimental (correlational) study What are the disadvantages Why donrsquot we see

more longitudinal research

C1 The use of cross-sectional research designs continues despite many calls for

replacing them with longitudinal designs (a) What are the main strengths of

cross-sectional designs (b) What weaknesses do they have that longitudinal

designs supposedly counter (c) What are the implications of cross-sectional

designs for theory development (Consider internal and external validity)

C2 A good number of research efforts on the micro (OBHR) side rely on cross-

sectional designs in which surveys are used to collect both IVs and DVs from the

same respondent What are the various problems associated with such a design

What changes could be made to this design to alleviate andor eliminate these

problems What are the downsides of the new design (add in conveniencenon

probability samples)

C3 What are some of the major strengths of a longitudinal design Why dont we see

more of them

C4 Assume that you are analyzing a large sample of longitudinal survey data TH

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Identify two characteristics of your data that would make ordinary least squares

regression (OLS) inappropriate for conducting analyses For each of these

characteristics describe an alternative analysis that is superior to OLS

D) Qualitative vs Quantitative Designs

D1 Compare and contrast the objectives strengths and weaknesses of

qualitativeobservational methods (eg ethnography) versus quantitative methods

(eg survey) in conducting empirical research on macro organizational issues Be

sure to consider issues of reliability generalizability validity ethics and

practicality in your answer Describe situations where one method or the other

may be most appropriate

IC

D2 In recent years a number of individuals have taken the position that qualitative

research is superior to quantitative research a Compared with quantitative

research what are the principal strengths of qualitative research b Compared

with quantitative research what are the principal weaknesses of qualitative

research

E) Observation vs Experiment

E1 Compare and contrast the objectives strengths and weaknesses of observational

methods (eg ethnographic methodology) and experimental methods Be sure to

consider issues of precision generalizability and realism in your answer Consider

also situations where one methodology or the other might be most appropriate

F) Laboratory vs Field Research

F1 Of the various research approaches available to organizational researchers

laboratory experiments are held up as an ideal because of their potential for high

internal validity On the other hand field research including participant observation

and field experiments have high potential for external validity Discuss specifically

the strengths and weaknesses of laboratory versus field experiments How would

positivist researchers think differently about these tradeoffs than would

phenomenologists

F2 In conducting an experiment in the laboratory or designing survey research there are

certain unique opportunities and limitations of each research strategy A) Describe a

hypothesized relationship between two constructs an independent and a dependent

variable in a substantive area of your choice (eg Strategy OB HRM

Organizational Theory) provide a verbal mathematical and graphical statement of

that hypothesis B) Briefly discuss how you would examine this hypothesized

relationship in a laboratory using experimental techniques versus a field survey using

correlation techniques C) Discuss what may be considered typical strengths and

weaknesses of each strategy regarding issues of precision and control in the

manipulation or measurement of the variables of interest realism of the context for

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

the participants and generalizability of the findings to other elements of a chosen

population D) Finally discuss advantages if any of using laboratory and field

survey methods in tandem

G) Experimental Design

2) A colleague suggests to you that only true experiments can provide useful results in

science Why might she say this What would you offer as a defense for the role of quasi-

experimental and non experimental designs in the context of the social science In

answering this question make sure you identify the defining characteristics of each design

type

9) Why and when should we be concerned about non response in research What are the

things we can do to try and decrease non response (or increase response rates)

11) Why might a researcher choose to utilize an experimental design that incorporates pretest

measure on the DV

G1 Experimental designs oftentimes come under attack due to perceived limitations in

external validity (a) Relative to other design types what strengths and weaknesses

do they offer (b) What experimental design structure is the strongest Why (c)

What is the relevance of research design for results and theory development

G2 Meta-analysts often investigate whether the quality of research design (eg

randomized experiments vs non-randomized designs) influences average effect

size In their meta-meta analysis of psychological educational and behavioral

treatments Lipsey and Wilson (1993) found that In a given treatment area poor

design or low methodological quality may result in a treatment estimate quite

discrepant from what a better quality design would yield but it is almost as likely

to be an underestimate as an overestimate (p1193) A few methodologists (eg

Shadish) have attempted to identify factors that may explain the relative magnitude

of effect sizes from randomized and nonrandomized experiments Please review

these findings and add your tentative model of the conditions under which random

experiments will produce larger smaller or equal effect sizes relative to

nonrandom designs Discuss possible implications for research design in the

context of real-world interventions

G3 Analysis of covariance and randomized blocks designs are sometimes referred to as

noise-blocking designs Explain what this means and give an example of (a) a

randomized blocks experiment and (b) an analysis of covariance experiment Keep

your examples simple Describe how such designs are noise-blocking when used

with randomized assignment of subjects to experimental condition and contrast

that with the role such designs serve in quasi-experimental designs

G4 Team performance in Bafko Industries varies considerably across teams and from

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

month to month The senior VP in charge of team development thought that a team

training program could improve the performance of these teams A consultant

convinced him that his firms training program which lasted only one week and

cost only $25000 per team) could raise the performance of Bafkos teams The VP

however wanted to experimentally test the efficacy of this training program before

he adopted it widely throughout the firm So the consultant suggested the following

experimental test

There were 48 teams in Bafko that showed significant variation from month to

month in their level of performance The consultant chose the 24 worst performing

teams based on last months data and assigned them to the training group After

all he argued these were the ones who most needed it The other teams were

assigned to the control group The training group received the training program

(lasting one week) at the beginning of the next month while the control group

teams did their normal routine Performance scores were gathered at the end of the

month The results showed that the training group teams had a significantly larger

rise (ie statistically significant) in their performance than did the control group

teams (who as a group did not improve at all) The consultant proudly displayed

these results and argued that they convincingly demonstrated the efficacy of his

program

How would you advise the VP Specifically what is the validity of the consultants

conclusions If you think he might be mistaken what specific threats to internal

validity are particularly plausible

G5 Consider the following design

N O Xa O

N O Xb O

N O Xc O

N O O where

Xa Xb and Xc represent different levels of a training program that varied in its

length across three different work groups In group a the training time was 40

hours in group b it was 30 hours and in state c it was 20 hours The training was

designed to instill customer service values in the team members O represents a

measure of customer service attitudes measured with a 20-item Likert scale This

measure was administered one week before the training program and 3 months

later The researcher hypothesized that the training program would improve

customer service attitudes and that longer training would have stronger effects

The design is a non-equivalent groups design because people were not randomly

assigned to groups even though groups were randomly assigned to treatment

condition Each group has a sample size of 36 members The reliability of the

customer service attitude scale was 78 at the pretest and 80 at the posttest The

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

pretest means were as follows

Group A 26

Group B 23

Group C 27

Control Group 25

a Construct a regression model that will test the researchers hypothesis Be

explicit here describing each variable in the model and what regression effect(s)

you would expect to find if the hypothesis were correct (Hint everything you need

to know is in Trochims chapter 11 and what we discussed in class) (10 points)

b Describe any serious threats to internal validity (10 points)

c Discuss how you might amend this design to improve its internal validity(10

points)

G6 Often in organizational research the researcher is faced with having only one group

on which to experiment and thus cannot use a control group when conducting

experiments This occurs for example in taxation experiments when you cannot

selectively apply new tax laws to different groups or when you wish to test the

efficacy of an incentive plan that must be applied to all members of a company In

such situations the researcher must be creative in assembling a quasi-experimental

design that has high validity (and usually the challenge concerns internal validity)

Consider the following scenario

A manager of a retail store wants to test the efficacy of an incentive program in

changing the selling behaviors of her retail sales clerks Specifically she wants to

test a behavioral reinforcement program in which sales peoples behavior is

recorded on in-store video cameras The tapes are analyzed each night and the

experimenter tabulates the frequency of various behaviors (eg offering assistance

to customers restocking items straightening up displays suggesting accessory

items to accompany the items that a customer is purchasing) The manager then

rewards the clerks with a monetary bonus based on the frequency of desired

behaviors She would like to conduct a valid experiment to see what effect such an

incentive program has on employee behavior but because of concerns about

employee morale she cannot assign clerks to treatment and control groups Instead

she has to apply the experimental treatment to everyone in the store She learned in

her research methods class (which the company requires all its managers to take)

that a single group experimental design typically has very low internal validity

Your job as a consultant is to help her design an experiment given her constraints

that would produce interpretable (ie internally valid) results Describe the details

of the design you would recommend using the notation in Trochim (2001) If the

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

design you come up with has a specific name use it As part of describing your

design discuss how the data would be analyzed and what particular effect you

would look for

G7 Why are experimental research designs held in such high regard

G8 Compare and contrast experimental quasi-experimental and correlational research

designs What characteristics separate these three designs What are their

advantages and disadvantages especially in comparison to one another

G9 What is the difference between a within-subjects experimental design and a

between-subjects experimental design When might a within-subjects design be

appropriate When might it be inappropriate

G10 Describe the difference between a basic within subjects and between subjects

experimental design What are the advantages and disadvantages of each

especially in comparison to one another Give an example from your area of

research in which each would be appropriate

G11 What are the major threats to the internal validity of a pretest - post test

nonequivalent control group design How can this design be improved with regard

to increasing a researchers ability to make causal inferences

G12 A production manager is interested in testing the hypotheses that participation in

quality circle programs will result in increased productivity and reduced

absenteeism A questionnaire is administered to a group of 92 employees

concerning their attitudes toward such programs and 46 with the most favorable

attitudes who are willing to participate are selected for the treatment group The

remaining 46 are the control group Measures of performance (unitsmonth for the

group) and absenteeism (lost daysmonth for the group) are kept over a three month

period during which the treatment group has weekly quality circle meetings The

results are as follows

Month 1 Month 2 Month 3 Month 1 Month 2 Month 3

Performance Absenteeism

Treatment 100 115 121 6 9 4

Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3 The only significant

difference is performance for the treatment group so the manager concludes that

quality circles raise productivity but do not have an effect on absenteeism What

are the primary threats to internal and external validity in this study What are

possible alternative explanations for the differences (or lack of differences)

between the two groups What additional information might help explain these

results

G13 One stream of current strategy research contains event studies In these studies

investigators basically correlate the reaction of a financial market to the presence

andor absence of the occurrence of some event at a firm (eg the resignation of a

CEO) In a different approach to such a study suppose that one investigator (Jean)

conducts a natural field experiment on the effects of competitor mergers on a

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

firms environmental scanning activity Through previous arrangement Jean has

been working with 5 medium-sized computer hardware and 5 medium- sized

software firms on the west coast Shes been collecting a paper trail of internal

memos reports and phone calls that might measure environmental scanning

activity Shes been finding that the firms in both industries do about the same

amount of scanning (no significant differences) However in the midst of her data

collection Norton announces a merger with Symantec (software) A year later

Jean compares the amount of post-merger environmental scanning in the software

industry with that in hardware industry Just as Jean had hypothesized a t-test

shows a greater amount of post-merger scanning among the software firms She

concludes that her hypothesis was supported mergers of competitors cause an

increase in scanning

Critique the study described above in terms of its design flaws Identify the design

Jean used Briefly allude to problems the study has with external validity but

concentrate on the Cook and Campbell threats to internal validity Provide a

rationale for why each threat is or isnt operating in the study Link each of the

relevant threats to the specific constructs being studied (mergers and environmental

scanning) Finally do a summary evaluation -- what can be learned from this

study

H) Quasi-Experimental Design

H1 The University of Mississippi observed that the graduation rate of students with poor

grades during their freshman year was much lower than those with better grades

Wanting to improve the chances of these low performing students they tested the

effects of a new rule for freshmen The rule stated the following At the end of their

first semester all students with a GPA between 200 and 225 must register for

EDUC 1003 during their second semester Students with a GPA lower than 200

were put on academic probation EDUC 1003 consisted of a course whose objective

was to help students develop good study skills The theory was that poor study skills

were a significant cause of poor academic performance and that study skills could

be learned

They wanted to evaluate the effectiveness of this intervention so they collected data

on the GPAs of students beginning with their first semester and continuing through

to their third semester At that point they wanted to analyze the data to determine

whether the new rule was making a significant difference in the students academic

performance

What is the best way to analyze these data to reach an interpretable conclusion

Describe your approach and then discuss the major threats to validity that it has

Their sample size of students was as follows

Total N= 3454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Number of students with first semester grades between 200 and 225 (and thus took

EDUC 1003) was 956

H2 A nationwide furniture company wants to determine if a diversity training

program will improve individual sales performance on the floors of its urban retail

outlets A group of 45 salespeople in one city were given two weeks of training

Measures of individual sales performance were taken once two months prior to the

program and once two months after Forty-five salespeople (who did not receive the

training) in a similar city had the same measures taken at approximately the same

points in time The training group increased their average weekly sales from $14000

to $16500 while the non-training group improved theirs from $13000 to $14500

The change for the training group is statistically significant the change for the non-

training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H3 An accounting firm is interested in determining if a re-training program will reduce

errors made by its audit teams A group of 45 auditors in the home office location

are given two weeks of re-training Measures of the frequency of detectable errors

are taken once two months prior to the program and once two months after Forty-

five auditors in similar office at a branch location have similar measures taken at

approximately the same points in time The training group reduced their average

number of errors per audit from 8 to 3 while the non-training group reduced theirs

from 11 to 9 The change for the training group is statistically significant the change

for the non-training group is not

How would the interpretation of these results in this quasi-experimental design

differ from a true experimental design What are the threats to internal validity in

the above study and how might they be operating (be specific) What additional

information and analyses would be useful in controlling for these threats

H4 Describe the main classes of threats to validity that are found in quasi-experimental

research designs

H5 Dr Delta an organizational development consultant was hired by a firm that has

600 pizza parlors nationwide to assist in implementing organizational changes that

he hopes will result in higher levels of job involvement At the outset of the study

subjects had jobs with very low levels of enrichment and were paid on an hourly

basis He designed an experiment in which stores were randomly assigned to one of

three treatment conditions Condition 1 subjects experienced job enrichment (JE)

and a change to performance-based pay (PBP) Condition 2 subjects experienced

only JE and Condition 3 subjects experienced only PBP The parlors were

sufficiently far apart from one another and communication among the parlors so

poor that subjects in any parlor were totally unaware of changes made in other

parlors At the end of 18 months he measured the job involvement levels of workers

and then tested for differences between the three conditions using a standard one-

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

way analysis of variance In view of Dr Deltas design answer the following

questions

How sound is Dr Deltas design Comment on the weaknesses of the design

I) Interrupted Time Series Design

I1 What threats to internal validity are and are not ruled out by an interrupted time series

design that includes the introduction of a treatment and its removal at a subsequent

time period List and explain In the process be certain that you provide a diagram

that illustrates such a design

J) Validity Some Questions are about the validity of a measure others about experimental validity)

10) Identify four threats to internal validity and briefly describe what they mean

J1 It was some time ago that the tripartite view of validity was abandoned in favor of a

unitarian view in which different evidentiary bases for validity might be put forward

Binning amp Barrett (1989) represent an attempt to place multiple evidentiary bases into a

single framework Cook amp Campbell (1979) describe four aspects of validity but they

arent really meant to represent evidentiary bases Adding to the confusion is the fact

that while Binning amp Barrett and Cook amp Campbell use some of the same terms these

two pairs of authors clearly mean different things by these terms and intend for them to

be used in different ways Your task is to 1) Describe the evidentiary bases discussed

by Binning amp Barrett 2) Describe the aspects of validity described by Cook amp

Campbell 3) Describe the similarities and differences between the two lists including

the different purposes served by the two lists and 4) Present a single framework that

incorporates both This last task is the most important of the four

J2 What are the types of validity How would each affect an empirical study IC

J3 Define both internal and external validity Which is more important (it is not okay to

say that they are equally important) Defend your reasoning

J4 Why might someone argue that the construct validity of a measure can never really be

established

J5 Describe each of the following forms of validity content convergent discriminant

criterion-related and construct How is each assessed How do they related to each

other Which is most important (Make sure you defend your answer)

J6 Define and explain 6 of the 8 types of validity listed below and describe the type(s) of

information each provides Choose one specific type of study as an example (eg

cross-sectional study experiment quasi-experiment simulation ethnography etc) and

compare and contrast the usefulness and importance of each type of validity for that

type of study

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7 Designing empirical research has been called making a series of trade-offs

Unfortunately decisions that are made to increase one desired quality (like internal

validity) can also decrease another desired quality (like external validity) For each of

the issues below discuss the concerns involved (ie the trade-off) and identify the

qualities that are being increased or decreased

a laboratory vs field study

b using a 5-item measure vs a 1-item measure

c using a student sample vs a sample of full-time employees

d using a sample of employees from one company of 5000 vs 500 employees from

each of 10 companies

e collecting your data using one method vs collecting your data using multiple

methods

J8 There are four major types of validity in social science internal validity external

validity construct validity and statistical conclusion validity Please address the

following three issues concerning validity

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Are there trade-offs that can or must be made among the four types of validity

d Describe how the concept of validity varies across the research traditions of

experimental vs observational research

IC

J9 The quality of research is often assessed by examining validity issues As you are

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

aware there are numerous typesforms of validity (eg internal and external validity

construct validity statistical conclusion validity) These types of validity address

different aspects of the research process For this question please address the following

issues

a What is the meaning of the four major forms of validity (ie internal and external

validity construct validity statistical conclusion validity)

b Should there be a priority among these forms of validity If yes why If no why not

c Does the concept of validity vary across research traditions (ie experimental vs

observational research) If yes why If no why not

J10 External validity has received considerable attention in the social and behavioral

sciences Some researchers argue we are too concerned with the external validity of our

findings while other researchers argue that we are too unconcerned with external

validity Several points of view have been developed to explain what is external

validity When answering this question please be sure to address at least the following

issues

a Compare and contrast alternative conceptualizations of external validity

b Some researchers have argued that the role of external validity is invariant across

research traditions (ie experimental vs observational research) Defend the

perspective that the role external validity is invariant across these two research

traditions In addition present the weaknesses of this perspective

J11 In designing a research study researchers must consider trade-offs between internal and

external validity Explain what is meant by internal validity and external validity What

is the nature of these trade-offs How would you evaluatedetermine which is more

important internal or external validity for a particular study Why is this such an

important issue in organizational studies

J12 A researcher conducted a quasi-experiment to test the effects of participative decision

making (PDM) on job attitudes of employees Employees from different departments in

a large organization were assigned to either a treatment group (PDM implemented) or a

control group (no PDM) Random assignment was not possible because of work

schedules coordination of tasks etc Thus the design can be considered an untreated

control group design with pretest and posttest and is represented as follows (where

O=observation X = treatment)

________________

O1 X O2

-------------------------

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

O1 O2

_________________

The results of the experiment are presented in the graph below [no graph provided]

The dependent variable is job satisfaction and is measured on a 100 point scale (with

100 being extremely satisfied) How amenable are these results to causal interpretation

What is (are) the major threat(s) to internal validity B if any B for this type of design

given these results What steps can be taken to assess the extent of this (these)

threat(s)

K) Data Collection Methods

14) Describe the various methods of data collection (eg survey interview observation etc)

available to a social scientist What are the advantages and disadvantages of each relative to

each other

K1 Within micro organizational research much criticism has been raised concerning

the reliance on self-report survey data What potential problems does self-report

survey research raise How big of a problem do you think this is

K2 There are a number of methods for collecting data (eg survey interview archival

etc) that are available to a social scientistorganizational researcher Discuss each in

terms of their strengths and weaknesses both on their own and in comparison to one

another

K3 Pick an important construct in your favorite area of management research for which

there is some controversy about how it should be measured or operationalized (eg

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

environmental munificence or growth need strength) Give the prevailing verbal

definition of the construct and analyze the measurement controversy from two

broad angles First discuss how the controversy may or may not stem from claims

about how one measure is better or worse than another because it (or the other) is an

(a) archival record (b) direct observation (c) social report or (d) self-report That is

what are the relative strengths and weaknesses of each of these types of

measurement for the construct youre discussing Second discuss the construct

validity of the most commonly used measure(s) of this construct To answer this

second part you need to discuss just what construct validity is and how it can or

cannot be demonstrated In the situation you describe what could be done to

demonstrate or improve the construct validity of existing measures

K4 Interviewing in some form has been the workhorse for data collection in a variety

of research traditions positivistquantitative ethnographic feministpost-modernist

and increasingly historical research While each tradition elicits information from

respondents the goals and processes involved in interviewing vary considerably

across research traditions Discuss the purposes of the interview the relationship

between data and theory the roles assumed by the interviewer and the respondent

and the relevant criteria for assessing the validity and reliability of the information

obtained In addition compare and contrast the use of the interview as a data

collection method in the positivistquantitative tradition with its use in one of the

alternative traditions

L) Common Method Variance

L1 Many of us have had concerns about the reviews of our survey research which

suggest that data analyses of surveys are biased because of common method

variance effects of individual differences such as negative mood negative

affectivity and social desirability and the reliability and validity of the constructs

we are attempting to measure Define how these problems affect a research issue in

organizational behavior and the steps that might be taken to minimize such potential

criticisms of papers based on analyses of self-reported data through survey

instruments

L2 In a series of papers by Spector (1994) Schmitt (1994) and Howard (1994)

published in the Journal of Organizational Behavior arguments were provided for

and against the use of self-report questionnaire measures One of the issues

discussed that could limit conclusions drawn from relationships observed using self-

report measures was method variance What is meant by method variance what are

its sources and what impact is it expected to have on research findings

M) Sampling

M1 The manufacturing census samples all plants of 250 employees or more For

plants of 5-249 employees it takes a size proportional sample Plants of 1-4

employees are not sampled What challenges does this sampling method provide

to researchers What could you do to account for these problems if you were

TH

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

studying organizational (not plant) growth

M2 New York State has approximately 250000 employees in over 70 state agencies

commissions and boards ranging in size from approximately 50 to over 5000

employees Assume that you wanted to draw a representative sample of 1000

employees Describe two ways that you could draw such a sample and briefly

describe the advantages and disadvantages of each of these sampling approaches

IC

M3 The director of a national non-profit agency with approximately 10000

employees located primarily in regional offices in each of the 50 states the

United States has approached you and asked you to develop a study

(organizational survey) of organizational culture within the agency The agency

is about to embark on a variety of change initiatives including developing team

structures and the study is meant to examine various elements of organizational

culture that are potential barriers to change The director has indicated that he

would like to have the data analyzed at both the individual and office level of

analysis Because of funding limitations the agency will only be able to send the

survey to 1000 employees The director has indicated that there are three ways

of sorting employees all employees listed alphabetically all employees listed

alphabetically by state and all employees listed alphabetically by region (there

are between five and ten regions per state) Given these three ways of sorting

employees describe at least three different sampling approaches and the

advantages and disadvantages of each

M4 In your role as Teacher of Organizational Studies you are asked to prepare a

tutorial for beginning doctoral students on sampling design Assume that this

tutorial is part of a basic class in research design and that students in this class

have had a basic statistics course up to and including multiple regression Your

tutorial should focus on briefly defining and explaining the differences between

probability and non-probability samples You should make sure to present and

describe several types of probability samples (eg random systematic

stratified etc) and non-probability samples (eg convenience snow-ball quota

etc) and discuss general principles for when to use a probability vs a non-

probability sampling design For each sampling design you should also provide

at least one clear example of a situation where this type of sampling design is

appropriate

N) Cross-Cultural Research

N1 Discuss the problems in conducting cross-culturalnational comparison in terms

of (1) conceptualization and (2) research design and data collection

Define the three levels of measurement invariance How would the problems that

you have identified in part (a) affect measurement invariance How would you

examine measurement invariance by confirmatory factor analysis

TH

N2 Explain why using national boundaries as independent variable in cross-cultural

studies may not be appropriate in terms of theory and research methods What

alternative ways would you recommend

IC

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

O) General

O1 In your readings research has been described as choices among conflicting goals

(desiderata) For instance when choosing among research strategies a researcher

must choose among the desiderata of precision realism and generalizability The

trade-offs among desiderata occur for all aspects of research that is for the selection

of theories the treatment of independent variables research strategies the selection

of the problem to study rules of evidence for determining the meaningfulness of a

research finding and even the frameworks that describe the research process

Discuss what you see as the major trade-offs for each of the following aspects of

research That is discuss what you see as the major strengths and weaknesses

(advantages and disadvantages) of each aspect

a The trade-offs between internal and external validity

b The manipulation of the independent variable vs the observation of the

independent variable

c Basic vs applied research

3 DESIGNING A RESEARCH STUDY

A) General (Research Design)

3) Why is control so important in research design What are the different ways it can be

achieved (give a brief description of each control method) What are advantages and

disadvantages of each

A1 It has become almost a truism to note that the choice of methods influences theory

and questions data and conclusions (the causal direction might also be reversed)

Briefly discuss how the methods used have influenced research in any two of the

following areas leadership power and politics organization culture organization

justice Compare and contrast as possible Suggest some potentially fruitful

methodological directions for future work in these same topical areas and dont be

shy about reflecting on broader issues of how the field of organizational behavior

constructs its realities Lastly tie your discussion and reflections back to your own

future as a researcher what lessons or guidance do you derive from your

understanding of these issues

A2 Characterize as fully as possible what you see to be the methodological approach or

approaches in strategic management research Is there a dominant method or a wide

variety Discuss the reasons for the situation What are the advantages benefits etc

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

of the approach(es) in use What assumptions (ontological epistemological

ideological) do they embody Are current efforts ideal or at least adequate Are

there other approaches that could be useful If so how might researchers be

encouraged to use these as well If you see no advantage to more variety in

methodological approach support that point of view

A3 Describe a research question or area of interest to you and why you think it is not

only of interest but significance Using your understanding of research methodology

describe two ways of pursing the question that are as divergent on as many

dimensions as you can imagine Describe the advantages and disadvantages of each

approach How strong a link or fit is there between a research question and a

method

A4 In his 1957 article entitled The two disciplines of scientific psychology Cronbach

discussed the assumptions goals and methods used in experimental methods and

correlational methods in psychology

a According to Cronbach what is the main difference between the two disciplines

b Discuss the strength and weaknesses of each discipline

c What if anything could be done to improve the situation

B) Designing a Research Study

B1 Cook and Campbell describe the experiment as the research design with the

greatest methodological rigor However given the constraints present in most

organizations I-O Psychologists cannot often conduct experiments in applied

settings and must settle for techniques with less control than experiments

Describe a study to investigate the relationship between participation in a

supervisor training program and work performance which could realistically be

conducted in an organization If an experiment is not possible describe how you

would go about maximizing your ability to draw valid conclusions from your

design

B2 The XYZ Co has recently developed a sexual harassment web-based training

program The developers claim the program will cut training time and costs and

will be as effective or more effective than classroom-based sessions They

believe that this will be due to the increased availability of the program and the

ability for participants to review program material on demand Before the XYZ

Co developers launch this product they want you to determine if their claims

have merit Explain how you would design a study for this purpose

B3 Research suggests that demographic diversity improves group creativity but

reduces group efficiency Some argue that demographic diversity is just a proxy

for value and experience diversity and that demographic diversity on its own is

meaningless Design a test to settle this argument

TH

B4 E-mail has become a common form of communication in organizations and is

used for both formal and informal communication Because it has become so

IC

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

popular some people complain that the frequent use of e-mail leads to

miscommunication because they receive communications in e-mail that should

be accomplished through face-to-face or telephone conversations where there is

an opportunity for interaction You are interested in looking at this

phenomenon You are most interested in describing how e-mail is used for both

formal and informal communication and in examining the effect of the e-mail

use on organizational communication effectiveness Design empirical research

to address those issues

Describe your study in detail thoroughly explaining the research design

sampling plan data collection procedures ethical issues and proposed data

analysis for your research design Be sure to discuss the internal and external

validity of your research Which validity threats are ruled out and which

remain

REMEMBER Be thorough Your research design should demonstrate that you

are a competent researcher who is knowledgeable about the proposed research

methodology or methodologies Therefore be sure that you carefully describe

and give technical explanations for each component of your research

B5 Choose a specific situation involving employee selection eg artists for an

advertising agency (a) What strategy would you use in selecting the selection

measures (b) How would you combine the measures eg which measures

would you combine in a compensatory manner and which is a noncompensatory

manner (c) How would you assume test fairness defining various measures of

the term (d) How would you make final selections (e) How would you

validate your procedure (f) What considerations would enter in if your

company were too small to provide validation data (g) What difference would

it make if you were developing a procedure for deciding whether trainees

should be promoted as opposed to making new hires Please note--emphasize

but do not limit yourself to statistical issues

B6 There is evidence in the marketing literature of a non-monotonic relationship

between the expectations that customers have for retail service and their

eventual level of satisfaction with that service That is as a customers

expectations for service increase so does their eventual service satisfaction --

up to a point After that point the greater the expectations the lower the

eventual satisfaction Develop a hypothesis that includes a moderator variable

that might account for this evidence Give some reasonable rationale for it State

your hypothesis verbally as an equation as a directed graph (circle amp arrow

diagram) and as an X-Y function chart Discuss how it would or would not be

possible or useful to test your hypothesis via a laboratory experiment In your

discussion cover how and why a researcher could (not) manipulate the

independent constructs involved (mode X) whether or not random assignment

to conditions (mode R) is possible (including how may conditions would be

necessary) which alternative constructs could or could not be controlled (mode

K) and which constructs should be measured (mode Y)

B7 Assume you are the chairperson of a committee that must evaluate

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

undergraduate programs at your university Your task as chairperson is to

devise an approach (a design a research strategy a set of measures) that will

allow you to evaluate undergraduate programs Make sure that your approach

can be used in the future and at other comparable universities When developing

your plan be sure to address at least the following issues

a the type of design you would select Compare and contrast the selected design

with another possible design

b the type of research strategy you would select Compare and contrast with

another possible strategies

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B8 Assume you are an organizational research design expert assigned to the

Strategic Planning Department of a large multi-national manufacturing firm

The Department has developed a new PC-based software package called

STRATPACK The software (and a one-week training course to learn how to

use it) is intended to improve the long-range planning ability of middle and

upper-level managers

Your job as research design expert is to devise research (a design a research

strategy a set of measures) that will allow you to evaluate the effectiveness of

the STRATPACK program Describe a research design for evaluating the

program Make sure that your research approach can be used in the future and at

all sites in the company When developing your plan be sure to address at least

the following issues

a discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

b the type of design you would select (eg a design where you manipulate the

independent variable versus observe the independent variable) Compare and

contrast the selected design with another possible design

c how your design deals with the major forms of validity Be sure to examine at

least internal external and construct validity

B9 Assume you are a consultant hired to evaluate the effectiveness of an industrial

safety program undertaken by a large manufacturing organization The safety

program is a behavioral one that provides rewards such as monetary bonuses or

extra paid vacation days for safety behaviors in terms of accident-free days

weeks etc Your task is to devise an approach (a research strategy a design a

set of measures and a general analysis strategy) that will allow you to evaluate

the safety program Because of the potential for the influence of many

extraneous variables on any safety program it is difficult to design research a

single cross-sectional study than can evaluate the impact of a single safety

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

program How would you design a study to evaluate the impact of the safety

program When developing your plan be sure to address at least the following

issues

a the type of research strategy you would select Compare and contrast with

another possible strategies

b the type of design you would select Compare and contrast the selected design

with another possible design

c discuss how your design and approach deals with the major forms of validity

Be sure to examine at least internal external and construct validity

B10 One of the most important areas of research is forecasting of human resources

needs based on strategic positions of organizations For example NY State must

carefully forecast its human resources needs as it proceeds with significant lay-

offs of state employees Future human resources needs are generally evaluated

by either statistical ie regression or simulation or clinical judgment i e

supervisors judgments methods

a Design a single study that addresses the empirical question of whether

statistical or judgmental methods are better for forecasting human resources

needs You may discuss a program of research but please be quite specific in

your description of the first study

b What are the critical methodological issues in doing this kind of research

c What are the critical applied (or practical) issues in doing this research

d Are the issues in b and c related

B11 In the past decade computinginformation technology has changed many

aspects of organizational life For example the technology can make increased

decentralization possible by improving communication between central offices

and field offices Computer networks and word processors can enhance group

empowerment by making it possible for team members to create and edit

written group products quickly even if the individuals are at different locations

Some have argued that computinginformation technology will foster the

paperless productive office nevertheless others argue that the technology only

increases paper and reduces office productivity You are applying to the

National Science Foundation for a grant to conduct research that examines the

impact of computing information technology on organizational decision

processes The NSF application procedure imposes five basic requirements that

your research design must fulfill in order to receive funding Note A very

significant factor affecting NSF grant decisions is a clear demonstration that the

researcher is knowledgeable about the research methodology used Write your

response to each of the following five grant requirements Be sure not only to

respond to the requirement but also to demonstrate your research

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

competence

1You must design a study that uses two different methods

(interviewobservation survey experiment or quasi-experiment) Describe your

two methods in detail and provide a rationale for the key elements (eg

sampling data collection proposed analysis) of your methods Be sure to

discuss and evaluate the internal and external validity of both methods

2 As part of your research design you must select a conceptual (latent) variable

central to your proposal (use centralization of decision making as your

variable) and describe how you will develop your own measure (You may not

simply use off the shelf instruments)

3 You must describe how you will test for the reliability of the measure

(centralization of decision making) you developed in step 2 The NSF insists

that you describe how you will evaluate both the temporal stability and the

internal consistency of your measure Be sure to discuss what temporal stability

and internal reliability mean and how they are related

4You must describe how you will assess the validity of the measure

(centralization of decision making) you developed in step 2

5 NSF requires a brief statement and defense of the philosophy of science

underlying your research Discuss your position and show how your research

design is consistent with your philosophy

B12 Many organizations in the United States economy have installed some form of

Empowered Teams as an alternative or supplement to traditional bureaucratic

structure Preliminary evidence of the effectiveness of empowered teams

however is mixed--some organizations have claimed success and others have

reported failures The evidence however is predominantly anecdotal--written

for trade magazines by the organizations themselves There are few rigorous

empirical studies assessing the effectiveness of empowered or self-managed

teams You are applying to the United States Department of Labor (USDOL) for

a grant to conduct research that examines the effectiveness of empowered

teams The application procedure of the USDOL imposes three requirements

that your research design must fulfill in order to receive funding Write your

response to each of the three grant requirements The importance of each section

is implied by the evaluation points allocated by the USDOL to each of the three

requirements given in parentheses

1 (50 points) Your study design for assessing Empowered Team effectiveness

must use two different methods (interview observation archival analysis

survey simulation experiment or quasi-experiment) Describe in detail and

thoroughly explain the sampling plan instrumentation data collection

procedures and proposed data analyses for both methods Be sure to discuss the

internal and external validity of both methods and how the two methods

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

complement one another

2 (30 points) As part of your research design you must select one conceptual

(latent) variable central to your proposal and describe how you will develop

your own operationalization of the measure (You may not simply use off the

shelf instruments) Use the concept of group morale to demonstrate your

operationalization methodology Be sure to explain very explicitly to the

USDOL how you will assess the validity and reliability of the group morale

measure you are developing

3 (20 points) The USDOL traditionally has funded research conducted only

within the general paradigm of logical positivism but now is more open to other

research paradigms as well Your grant proposal must provide a statement and

defense of the explicit philosophy of science underlying your research Describe

your philosophy of science and show how your research design is consistent

with your philosophy

REMEMBER Be thorough You not only must design effective research but

you also must demonstrate to the USDOL that you are a competent researcher

It is absolutely vital to the USDOL that the principal researcher is

knowledgeable about the proposed research methodologies Therefore be sure

that you always carefully describe and give technical explanations for each

component of your research design and explain all technical terms (eg

Cronbach alpha KR-20 Likert scale)

B13 Assume that you are working as part of a research team that is responding to a

RFP (Request for Proposals) from a funding agency for research on work

socialization and newcomer adjustment The RFP calls for rigorous field

research that tests the effectiveness of theory-based interventions aimed at

facilitating the adjustment of new organizational members to complex andor

stressful work conditions The major goal of your teams research is to test the

effectiveness of a self-regulation intervention which trains newcomers to

identify and use relevant social information and cues to regulate work behavior

Your group hypothesizes that self-regulation training will be more effective

than other common treatments or interventions (eg realistic job previews

institutionalized socialization tactics)

You have been included in the team because of your methodological expertise

Your role is to design a rigorous field experiment or quasi-experiment to test he

effectiveness of the self-regulation training program REMEMBER THE

MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH

DESIGN Design a field study to include in the grant proposal Your proposal

should contain the following two sections

1 Study Design Construct and describe in detail an experimental or quasi-

experimental design to test the effectiveness of your intervention Describe in

detail the experimental groups (including how they will be constructed) the

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

research setting the sampling plan for study participants the treatment

manipulation and the data collection procedures Also indicate any control or

nuisance variables that may be measured

After describing the procedures write a justification of the design What threats

to validity does it rule out or limit and what threats to validity remain

II Measurement Describe the steps you would take to achieve valid and

reliable measure of the latent (conceptual) variable worker adjustment Your

team feels that as part of the proposal you must develop an original

operationalization of adjustment (thus you cannot simply use existing off the

shelf measures) Indicate in adequate detail how you would operationalize the

measure and indicate how you would demonstrate reliability and validity

B14 In describing their notion of the collective mind Karl E Weick and Karlene H

Roberts note Our focus is at once on individuals and the collective since only

individuals can contribute to a collective mind but a collective mind is distinct

from an individual because it inheres in the pattern of interrelated activities

among many people (p 360) Collective mind is not indexed by within-group

similarity of attitudes ideas or activities A collective mind is one in which

many people or many groups integrate their different attitudes ideas and

activities heedfully As Weick and Roberts note People act heedfully when

they act more or less carefully critically consistently purposefully attentively

studiously vigilantly conscientiously pertinaciously (p 361)

The idea of the collective mind as heedful interrelating is developed in the

following excerpt When people make efforts to interrelate these efforts can

range from heedful to heedless The more heed reflected in a pattern of

interrelations the more developed the collective mind and the greater the

capability to comprehend unexpected events that evolve rapidly in unexpected

ways When we say that a collective mind comprehends unexpected events we

mean that heedful interrelating connects sufficient know-how to meet situational

demands For organizations concerned with reliability those developments

often consist of unexpected non-sequential interaction (p366)

An example of heedful interrelating occurs as a pilot lands herhis aircraft on the

deck of an aircraft carrier The pilot does not actually land She is recovered

A recovery is an interrelated set of activities among air traffic controllers

landing signal officers the control tower navigators deck hands the helmsman

driving the ship etc These activities can be interrelated more or less heedfully

depending on the care with which the different units contribute represent the

situation and subordinate their activities to the single purpose of successful

recovery (p 363)

You are to design a two-part study

Part I You are to design a measurement study which assesses heedful

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

interrelating (ie collective mind) with a quantitative measure in a replication

of an aircraft carrier flight deck or other situation of your choosing Describe

your research strategies including operationalization of the variable selection of

the sample subjects data collection data analysis and so on for the purpose of

developing a measure of heedful interrelating

Part II Using the measurement tool developed in Part I and any other tools as

needed describe how you would design a study which would compare heedful

interrelating across organizations Choose at least one macro-organizational

variable (eg technology environment) as your basis of comparison Here you

must describe how you would make your selection of organizations to be

compared your data collection procedure data analysis and so on

B15 Many organizations in the United States have installed cross-functional teams to

counteract the silo mentality of traditional bureaucratic structure Preliminary

evidence of the effectiveness of cross-functional teams however is mixed --

some organizations have claimed success and others have reported failures The

evidence moreover is predominantly anecdotal--written for trade magazines by

the organizations themselves Since there have been few rigorous empirical

field studies assessing the effectiveness of cross functional teams the National

Science Foundation has decided to fund research on the topic

Assume that you are submitting a research proposal to NSF for a grant to

research the effectiveness of cross-functional teams Describe in detail and

thoroughly explain the research design sampling plan data collection

procedures ethical issues and proposed data analysis for your research design

Be sure to discuss the internal and external validity of your research Which

validity threats are ruled out and which remain NSF also requires that you

provide a statement and defense of the explicit philosophy of science underlying

your research Describe your philosophy of science and discuss how your

research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to NSF that you are a competent researcher It is

absolutely vital to NSF that the principal researcher (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research design and explain all technical terms (eg Cronbach alpha KR-20

Likert scale)

B16 Assume that you are submitting a research proposal to the Department of

Commerce (DoC) for a grant to research the effectiveness of virtual work teams

A virtual work team is a leaderless collection of individuals who are

geographically separate from one another and share their expertise using

internet media such as a listserver or Lotus notes The DoC requires that the

researcher use both qualitative and quantitative methods to gather data Describe

in detail and thoroughly explain the research design sampling plan data

collection procedures ethical issues and proposed data analysis for your

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

research design Be sure to discuss the internal and external validity of your

research Which validity threats are ruled out and which remain The DoC also

requires that you provide a statement and defense of the explicit philosophy of

science underlying your research Describe your philosophy of science and

discuss how your research design is consistent with your philosophy

REMEMBER Be thorough You not only must design effective research but

you must also demonstrate to DoC that you are a competent researcher It is

absolutely vital to DoC that the principal investigator (you) is knowledgeable

about the proposed research methodologies Therefore be sure that you always

carefully describe and give technical explanations for each component of your

research

4

CRITIQUE OF AN EMPIRICAL ARTICLE

A1 In your role as Critic of organizational studies research you are to evaluate an article

submitted to a journal and to describe in detail the weaknesses and the strengths of

the research Critique the attached article An Exploratory Hypothetico-Deductive

Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations by an

anonymous author submitted to the Journal of Organizational Research The format

for your critique should list the strengths and weaknesses of the research and give a

clear rationale for all your criticisms and suggestions In addition it is important to

make specific recommendations for improving the research and the manuscript You

should explain to the author how and why you would redesign the research and

reanalyze the data to improve it Finally indicate your acceptance or rejection of the

manuscript for publication Be sure to explain and justify your criticisms and

recommendations In your review be sure to discuss

1 The appropriateness and adequacy of the literature review

2 Importance of the problem

3 Conceptual development and support of the hypotheses

4 Appropriateness of the design for addressing the purpose of the study

5 Soundness of research methods

6 Adequacy of data analysis

7 Legitimacy of the data interpretation and conclusions

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

8 Quality of the discussion

A2 Please evaluate the internal external construct and statistical conclusion validity of

[some recent article] Provide an argument given the current state of the literature

on this topic for which validity (or set of validities) is most critical at this juncture

in the field Finally given your critique and assessment design a follow-up study

that improves upon the key validity (or validities)

5 MEASUREMENT

A) Measurement Terms

A1 There are four levels of measurement rigor in management research Discuss each

one What properties do measurements at each level have (ie how much

information do the numbers from each level of measurement convey) How can the

numbers be changed (transformed) and still retain the properties of their

measurement level Pick one construct from a favorite research domain and give an

example of an operationalization of that construct at each of the four levels of

measurement rigor (provide arguments about why you think those examples reflect

each level) Lastly treat each examplelevel as a dependent variable in some

statistical analysis Given an example of a statistical technique appropriate for

analyzing data at that level of measurement rigor

A2 Define (and discuss important features or issues concerning) the following

constructs measurement (Define any six terms)

1 Construct validity 6 Item analysis

2 Likert scaling 7 Semantic differential

3 Cronbachs alpha 8 True score

4 Unobtrusive (trace) measurement 9 Generalizability theory

5 Interval scale 10 Criterion-related validity

11 Sociometric choice measures

B) Operationalization

B1 a) You are conducting a study which requires you to measure Hong Kong real

estate agents self-assessment of performance Since there is no existing

measurement available you have to develop the instrument by yourself

Describe how you would develop the measurement assuming that you have no

idea of how many dimension the construct has

b) Suppose you have developed an instrument consists of 20 questions

measuring 4 dimensions (5 items per dimension) from (a) You collected data

from another 300 real estate agents to validate your instrument Explain how

TH

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

you would use structural equations model (SEM) to examine the reliability and

validity of the instrument What are other concerns about reliability and validity

that are not addressed by this approach

B2 The research and business world have been fascinated with the concept of

organizational culture However operationalizing this construct has been a

difficult task Discuss the ways in which the construct has been operationalized

(eg technique static v dynamic sample selection generalizability) Critically

analyze each of these methodologies Then choose a methodology you would

use to examine organizational culture and defend your choice Describe how

you would study culture and what more would be known about culture as a

result of your study

B3 Suppose you are setting out to develop a new measure of a construct You want

this measure to be used in survey research and you want to establish its validity

and reliability Your goal is to have at the end of the process a measure that is

defensible and likely to be widely accepted by other researchers studying that

construct What steps would you take to conduct such a program of research

For each step explain the purpose of that step explain why that step is

important explain what methods and statistical procedures you would employ at

each step and explain what evidence you would hope to gather at each step

B4 Assume you are interested in constructing a new measure of (fill in with a

relevant variable from your field) Detail the major concerns that need to be

addressed and the steps that need to be taken in working toward validating this

new measure

B5 Assume that you recently read an article on workplace violence which you

found very interesting The authors argued that one of the predictors of

workplace violence was a construct called propensity to be violent and used a

one-item measure as follows How likely are you to become violent when

provoked You had serious doubts about the authors measure and decided to

develop your own

(a) How would you go about developing items to measure propensity to be

violent

(b) How would you determine whether the instrument was reliable

How would you determine whether the instrument was valid

B6 A concern in organizations is retirement of baby boomers within the next 5 -

10 years This concern is focused on the loss of older employees with skills

critical to the core business functions of organizations In order to continue to

have employees with those critical skills it seems necessary to retrain some

older workers However past practice in most organizations has been not to

retrain older workers for a variety of reasons mostly financial A researcher

working for the Human Resources Department of a large corporation - 13500

employees in 12 different plant locations - hypothesizes that the major reason

for not retraining older workers is ingrained in the culture of the corporation

IC

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

The researcher feels that older workers are not as highly valued as younger

workers in terms of their potential usefulness after retraining The researcher

searched the literature but found nothing in the existing literature to measure

the cultural value construct What should the researcher do to establish a reliable

and valid self-administered paper and pencil measure of this construct namely

the value of retraining older workers

a Please describe and explain the steps in temporal order that the researcher

must follow to develop a measure

b How would you evaluate the measure based on the technical requirements of

reliable and valid measurement as well as any practical issues in using the

measure considering the size of the employee population

B7 Assume that you are going to conduct empirical research to assess the role that

the complexity of an organizations manufacturing technology plays in

determining the level of centrality of decision making in the organization

Describe and justify the steps you might follow to develop an operationalization

of the concept of technological complexity What standards would you use to

determine if your operationalization is valid

B8 Organizational researchers have several strategies available to them for

gathering information (eg self-report measures archival measures direct

observation and trace measures) Assume that you are going to conduct

empirical research to assess the role that the structure of the organization plays

in determining the level of organizational effectiveness Briefly review the

strengths and weaknesses of the basic strategies (that is self-report measures

archival measures direct observation and trace measures) for measuring

organizational effectiveness Next describe and justify the steps you would

take to develop an operationalization of the concept of organizational

effectiveness What standards and procedures would you use to determine if

your operationalization is valid

B9 One of the most important topics of recent research has been stress in the work

place One of the most important problems with understanding this literature is

the varying definitions and measurement of this construct Suppose you had

decided to investigate the relationship between stress at the work place and dual

career family happiness

a Since there are multiple ways to measure stress in the work place describe

two ways to do this Hint - avoid using two self-report measures

b Describe how you would decide on the best single measure In doing so

describe the steps you would take in evaluating your measure Be sure to define

your meaning of validity and reliability It is important that your measure have

all the necessary psychometric characteristics so that other scientist can trust the

results of your research

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

c Be certain that you answer the question Validity for what

d In order to understand the results of your study you need to compare your

measure of stress at work with other measures of this construct How would you

do that

B10 Assume that you have been hired by a national survey research company

Shash Inc to write items that will be used in a national survey of attitudes

toward capital punishment There are a number of criteria that can be used to

guide your item writing What are they Briefly describe five (5) such criteria

B11 Assume that you are going to conduct empirical research to determine the

impact of interpersonal trust on organizational decision making Describe and

justify the steps you might follow to develop an operationalization of the

interpersonal trust concept

B12 Assume that you are going to conduct empirical field research to assess the role

that employee empowerment plays in determining the level of organizational

commitment of employees Describe and justifyexplain the steps you would

follow to develop a valid and reliable operationalization of the concept of

employee empowerment

Explain the procedures and standards you would use to determine if your

operationalization is valid and reliable

How would you assess the construct validity of such a measure

Discuss issues concerning the level of observation of your measure and level of

analysis of the data

B13 Assume you are interested in developing a new scale to measure construct X

Further assume that you have used previous literature and your own conceptual

development to delineate 2 dimensions of X specifically XF1 and XF2

Describe the steps you would go through in developing testing and evaluating

this new scale With regard to testing and evaluating be sure to consider the

role of factor analysis (both exploratory and confirmatory) and item analysis

Also you will want to make sure that your new multidimensional scale shows

evidence of validity (several kinds) and reliability so be sure to examine these

issues and explain how specific information gathered from the above testing

contributes to these concerns

C) Reliability

4) What is reliability and why is it so important What are the different ways reliability can be

assessed and when is each appropriate

15) What does coefficient alpha assess What are the ways coefficient alpha can be increased

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

C1 Describe and discuss the concept of reliability and all of its various forms

including their assessment When is each form appropriate and what information

is provided by each index

C2 In establishing congruence between multiple measurements we often compute

indices of reliability andor indices of agreement These two types of indices

address very different questions Your task is to 1) Define reliability and

agreement 2) Explain when each is useful using examples in which one but not

the other is appropriate and 3) Briefly describe two reliability indices and two

agreement indices

C3 Explain reliability in terms of the classical test theory What are the

consequences of unreliable measures in hypothesis testing What are the factors

that may affect the reliability of a multi-item scale

IC

C4 Generalizability theory is used within IO psychology more often today

However most researchers continue to use Classical Test Theory

conceptualizations of error to determine reliability What are the implications of

using classical test theory versus generalizability theory to assess reliability

C5 In establishing congruence between measurements we often compute indices of

reliability andor agreement Define reliability and agreement Explain the

differences in reliability and agreement and describe under what conditions

assessing reliability is more appropriate and under what conditions assessing

agreement is more appropriate

C6 Define measurement reliability in terms of a specific measurement theory (eg

true test score theory domain sampling theory generalizability theory IRT)

Discuss the different approaches for estimating reliability and the contexts in

which each might be appropriate Use examples throughout

C7 What are the consequences of unreliability in a measure

C8 What are some of the conditions that might keep the internal consistency (eg

alpha) of a given measure low Is a low alpha always a sign of a problematic

measure

C9 Define reliability Name some methods of calculating reliability How well do

these methods map up with the definition of reliability What is the impact of the

number of test items and sample size on Chronbachs alpha and why would this

effect occur or not occur (Note Do NOT discuss inter-rater reliability)

C10 Schmidt and Hunter (1996) described several types of measures typically used in

organizational research and identified the type(s) of measurement error affecting

each of them in various situations For each measure below indicate the types of

error and sources of error that are most likely to affect it Then indicate what

could be done to reduce errors of measurement andor what can be done to

control for errors statistically

a Supervisor ratings of subordinates job performance

b Self-ratings of ones own job performance

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

c Self-reports of ones own job-related attitudes

d Sales figures for a salesperson (or production quantity for a production

worker)

d Aptitude tests taken as part of the hiring process

C11 Suppose you had conducted research on the relationship between organizational

commitment and rigidity of organizational structure in 200 firms Your measure

of rigidity was based on number of organizational authority levels and the degree

to which written organizational rules existed and defined role behaviors for

employees Thus an organization with a large number of organizational levels

and specific written policies defining appropriate role behaviors for employees

would score high on rigidity of organizational structure After analyzing your

data you find that there is a statistically significant negative correlation between

commitment and rigidity which is consistent with your hypothesis However the

reliabilities for your two measures are

(1) commitment = 23 (2) rigidity = 83

How should you interpret your results What could you do to fix this situation

Describe the specific steps (in order) you would follow to develop a more

reliable measure of commitment

C12 (a) Define the concept of reliability (of measures) in a clear concise and

accurate manner (b) Provide a formula that expresses the reliability of a measure

in terms of two components of variance ie total variance and the error

variance

D) Multi-level Assessment

D1 One of the major challenges for group-level research is the issue of aggregation of

individual data to the group level

(a) Under what conditions (eg types of data number of raters) should the various

interrater agreement and reliability methods be used (Rwg Kappa and Weighted

Kappa percent agreement average pairwise correlation ANOVA methods

(interclass correlation coefficient WABA)) What are the advantages and

disadvantages of each of the methods

(b) Explain the controversy between Frank Schmidt Larry James and others

regarding the differences between interrater agreement and interrater reliability

Who is right Do the differences matter in research practice

Defend your answers

(c) Review two articles from the organizational behavior literature that have had to

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

deal with aggregation issues Critique the methods the authors used to make

aggregation decisions

(d) Provide some advice to researchers regarding what they should do if interrater

agreement and reliability indices indicate that it is not legitimate to aggregate their

data

(e) Provide some ideas for avoiding the need to aggregate in group-level research

through creative research methods and measurement strategies

D2 Cross-levelmulti-level research is becoming more evident in the organizational

research literature This type of research raises many issues and presents many

challenges in terms of designing the research measuring variables and collecting

and analyzing the data Assume that you are interested in aggregating data (eg

from the individual to the group or from the firm to the industry) for one or more

variables What are the major issues you will need to deal with

D3 A researcher wants to measure organizational climate for continuous learning

Organizational climate is operationalized as a shared consensual aspect of the

organization -- the shared perception among employees that the organization values

and supports continuous learning Describe how you would measure organizational

climate in this case Specify (a) the level (ie individual group organization) at

which the construct exists (b) the level at which you would measure the construct

and (c) the procedures that you would use to establish that measurement is aligned

with theory (ie the construct validity of the measure)

E) General (Measurement)

E1 Studies in organizational behavior research often rely on single-item measures of

key constructs The use of single-item measures has long been criticized based

on the premise that single-item measures are unreliable However Wanous

(Wanous amp Hudy 2001 Wanous amp Reichers 1996 Wanous Reichers amp Hudy

1997) has recently argued that single-item measures can exhibit adequate

reliability Wanous based this conclusion on a procedure for estimating the

reliability of single-item measures using the classic formula for correction for

attenuation Subsequent studies have yielded mixed results regarding the

conclusions of Wanous (eg Loo amp Kells 1998 Nagy 2002) and the veracity

of the perspective advocated by Wanous remains in question Summarize and

assess the arguments for and against the use of single-item measures and

evaluate the procedure and evidence offered by Wanous regarding the reliability

of single-item measures Based on your assessment and evaluation develop

recommendations regarding the use of single-item measures in organizational

behavior research

Gardner D G Cummings L L Dunham R B amp Pierce J L (1998) Single-

item versus multiple-item measurement scales An empirical comparison

TH

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Educational and Psychological Measurement 58 898-915

Loo R amp Kells P (1998) A caveat on using single-item measures Employee

Assistance Quarterly 14 75-80

Nagy M S (2002) Using a single-item approach to measure facet job

satisfaction Journal of Occupational and Organizational Psychology 75 77-86

Wanous J P amp Hudy M J (2001) Single-item reliability A replication and

extension Organizational Research Methods 4 361-375

Wanous J P amp Reichers A E (1996) Estimating the reliability of a single-

item measure Psychological Reports 78 631-634

Wanous J P Reichers A E amp Hudy M J (1997) Overall job satisfaction

How good are single-item measures Journal of Applied Psychology 82 247-

252

E2 Ipsative measures have become prevalent in organizational behavior research

Ipsative measures ask respondents to answer questions relative to one another

such that the scores for all questions sum to the same constant for all respondents

Examples of ipsative measures include rankings Q-sorts and the allocation of a

fixed number of points (eg 100) across a set of response options Ipsative

measures may be contrasted with absolute measures which ask respondents to

answer questions independently from one another Absolute measures are

exemplified by rating scales using an interval or ratio numeric response format

The relative merits of ipsative and absolute measures have been debated for

decades (eg Cornwell amp Dunlap 1994 Hicks 1970 Johnson Wood amp

Blinkhorn 1988 Saville amp Wilson 1991 1992) yet the fundamental issues

underlying this debate have not been adequately addressed in organizational

behavior research For example measures of organizational culture and values

are often ipsative and some researchers advocate these measures over absolute

measures (eg Meglino amp Ravlin 1987 OReilly Chatman amp Caldwell 1991

Ravlin amp Meglino 1987 see also Rokeach amp Ball-Rokeach 1989) Review the

relative merits of absolute and ipsative measures Based on your review which

form of measurement is most appropriate and why Do your conclusions depend

on the construct being measured If so what types of constructs are best suited to

ipsative measures and what types are best suited to absolute measures Next

apply your conclusions to the measurement of culture and values in

organizations What specific arguments have been advanced in favor of ipsative

measures of culture and values How sound are these arguments Should ipsative

measures be used in the study of organizational culture and values If not how

can absolute measures of culture and values be developed that address the

concerns voiced by advocates of ipsative measures Provide a specific example

of the type of measure you would propose

Cornwell J M amp Dunlap W P (1994) On the questionable soundness of

TH

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

factoring ipsative data A response to Saville amp Willson (1991) Journal of

Occupational and Organizational Psychology 67 89-100

Hicks L E (1970) Some properties of ipsative normative and forced-choice

normative measures Psychological Bulletin 74 167-184

Johnson C E Wood R amp Blinkhorn S F (1988) Spuriouser and supriouser

The use of ipsative personality tests Journal of Occupational Psychology 61

153-162

Meglino B M amp Ravlin E C (1998) Individual values in organizations

Concepts controversies and research Journal of Management 24 351-389

OReilly C A Chatman J A amp Caldwell D F (1991) People and

organizational culture A Q-sort approach to assessing person-

E3 In organizational research measures often consist of items that represent different

facets of a concept Although such measures are prevalent and in some cases are

advocated (eg Law et al 1998 1999) they can present a variety of conceptual

and methodological problems Describe the specific problems associated with

measures that use items representing different facets of a concept Compare these

problems with the purported advantages of such measures Conclude by

indicating whether and under what conditions such measures are useful for

organizational research

Law K S Wong C S amp Mobley W H (1998) Toward a taxonomy of

multidimensional constructs Academy of Management Review 23 741-755

Law K S amp Wong C S (1999) Multidimensional constructs in structural

equation analysis An illustration using the job perception and job satisfaction

constructs Journal of Management 25 143-160

TH

E4 What is domain sampling and what are its implications for measurement of

constructs Compare and contrast the influence of domain sampling on the

following

a Confirmatory factor analysis studies

b Experimental research

c Qualitative field research

d Quantitative field research

E5 Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of

AFFECTIVE CONFLICT (AC) in decision-making teams She gets access to a

variety of outcomes assessment committees at each of 87 different schools in

the Dallas Independent School District and gains permission to tape-record two

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

of their meetings (one in February and one in March) Such committees have

been formed to develop quantitative ways to track pursuit of school goals and

provide accountability to local and state governments These are often divisive

teams with heated meetings because what gets assessed is what usually gets

rewarded After both of these meetings the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how

much AC occurred (eg People in our group often get angry with each other)

The individual scores are summed across items then averaged across team

members to come up with a group-level AC score for each of the 87 2

meetings The same procedures are followed for a 5-item measure of Group

Cohesiveness (GC) which is predicted to be reduced by higher levels of AC

Roberta also listens to every tape and from her expert judgment she rates each

groups meetings on the same 20 items Those 20 ratings are summed for each of

the 87 2 meetings Now Roberta is interested in evaluating her measure She

wants it to have excellent psychometric properties including reliability and

construct validity

a What is reliability Provide both a formal and informal definition Connect that

definition to the context given above

b How could you estimate the reliability of this new measure of AC Cover at

least three kinds of estimates and the sources of error each one assumes or is

estimating Pick one of those kinds of estimates and provide a rationale for using

it

c What is construct validity Discuss how the (un)reliability of the AC measure

could affect the conclusions one would draw from a study involving it and GC In

your discussion be sure to go over the connection between reliability and

validity and the connection between reliability and the probability of Type I or

Type II statistical errors

d What is common method variance How does it complicate the picture

described in (c) Be specific about how it might affect the statistical errors

mentioned above

6

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

6) What is the ldquoomitted variable problemrdquo and what are its statistical and theoretical

consequences How can you deal with this problem inn your own research

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

7) Discuss the difference between probability and non probability sampling What are the

advantages and disadvantages of each relative to the other

A1 The omitted variable problem is (or should be) widely known and most estimation

techniques are subject to this problem Describe the problem and discuss its

theoretical and statistical consequences What do you plan to do to deal with this

problem in your own research Can one ever fully address this concern in a given

research context Why or why not

A2 Describe and compare the information provided by each of the following (a) the

size of the difference between the mean scores (or the value of a given score in

terms of a particular scale of measurement) (b) the level of the statistical

significance obtained (c) the size of the confidence interval surrounding some

summary statistic (d) the proportion of variance accounted for What conclusions

can be drawn from each of these statistics in interpreting results of research How

do they relate to the design of the research

A3 The following eight terms come from standard statistical analyses Pick six of the

eight and define them Discuss important features or issues concerning each term

1 Main effect 5 Bonferroni correction

2 Interaction effect 6 Unstandardized regression coefficient

3 Type II error 7 Statistical power

4 Oblique rotation 8 Generalized (ordinary) least squares criterion

(Other possible terms

1 Main effect 7 Orthogonal comparison

2 Interaction effect 8 Omega-squared

3 Biserial correlation 9 F-test

4 Varimax rotation 10 Degrees of freedom

5 Type I error 11 Covariance

6 Statistical power 12 Non-parametric statistics)

B) Statistical Significance Testing and Power

8) What is statistical power and why is it important What are some ways to increase it Can

you ever have too much power

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

B1 Describe the following with respect to statistical power a) define the concept b)

list the factors that affect power and describe how they affect power c) describe

the steps required to plan research with respect to power taking care to mention

how the factors that you just described are included in the plan and d) describe

how power is increased through the use of repeated measures

B2 In recent years (and in not so recent years for that matter) methodologists have

criticized the fact that psychologists rely heavily on significance testing What

are the essential elements of these critiques What are some remedies to the

problems engendered by significance testing and why have these remedies not

been widely adopted

B3 (a) There is a debate among the IndustrialOrganizational psychologists on

whether we should abandon significance testing and report confidence intervals

instead Please summarize arguments from both sides and state your position on

this issue

(b) From time to time some researchers want to confirm null hypotheses in their

studies What are the concerns of accepting a null hypothesis What precautions

we can do to provide a stronger support for a null hypothesis

TH

B4 Explain the recent controversy about significance testing Critique arguments for

each side of the issue If you were a big-wig in the APA what changes if any

would you recommend in the policies regarding significance testing in APA

journals Support your answer

B5 Discuss the debate regarding statistical significance testing in psychology What

are some of the criticisms of significance testing What is your opinion on these

criticisms and on significance testing in general Support your argument

B6 What is statistical power Why should we as researchers and consumers of

research be concerned with it What are some consequences of low power

What are some consequences of high power What affects the power level of a

statistical test What can be done to increase power in various research designs

(experimental quasi-experimental and correlational)

B7 There has been an accusation made that empirical researchers depend too heavily

on significance levels to interpret research The accusers argue among other

things that researchers have inappropriately sanctified the p lt 05 level of

significance Describe and critique the arguments for and against the use of

statistical significance testing in empirical research What information other than

significance levels can be used to interpret the meaning of empirical research

C) Statistical Conclusion Validity

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

C1 Assume you have hypothesized a relationship between variables X and Y conducted

a study to investigate this relationship and found it to be statistically non significant

What are the various reasons that might explain why the X-Y relationship was not

significant

C2 What is statistical conclusion validity and what factors affect it Indicate the nature

of the effect on statistical conclusion validity of each factor you mention What can

researchers do to improve statistical conclusion validity

C3 In general what canshould a researcher do to maximize the likelihood that his or

her statistical conclusions will be valid Indicate the nature of the effect (on

statistical conclusion validity) of each factor that you mention

D) ModeratorsMediators

D1 Moderation and mediation provide different ways to model third variable effects

(a) Please define both of these terms (b) Sketch a conceptual model to show how

each relates to main effects (c) Using variables of your choice please write

hypotheses to reflect both (d) Explain how each is tested analytically

D2 Discuss moderators and mediators Provide an example of how (1) a moderating

variable influences the relationship between the independent and dependent

variable and how (2) a mediating variable influences the relationship between

the independent and dependent variable Describe how you would test for both

moderation and mediation using regression

D3 Many theories in organizational research predict that the effects of one variable

on another are moderated by a third variable Such moderator effects are

typically tested by entering the product of two variables as a predictor after

controlling for the two variables that constitute the product Several researchers

have pointed out that this product is often highly correlated with the squares of

the two constituent variables and have suggested that moderating effects may be

tested after controlling not only for the two constituent variables but also their

squares (eg Cortina 1993 Ganzach 1998 Lubinski amp Humphreys 1990

MacCallum amp Mar 1995) Discuss the strengths and weaknesses of this

approach as compared to methods that use only the product term and its

constituent variables Conclude with recommendations as to when and under

what conditions organizational researchers should test moderator effects with the

squares of the constituent variables in the equation

Cortina J M (1993) Interaction nonlinearity and multicollinearity

Implications for multiple regression Journal of Management 19 915-922

Ganzach Y (1998) Nonlinearity multicollinearity and the probability of Type

II error in detecting interaction Journal of Management 24 615-622

Lubinski D amp Humphreys L G (1990) Assessing spurious moderator

effects Illustrated substantively with the hypothesized (synergistic) relation

TH

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

between spatial and mathematical ability Psychological Bulletin 107 385-393

MacCallum R C amp Mar C M (1995) Distinguishing between moderator and

quadratic effects in multiple regression Psychological Bulletin 118 405-421

D4 Assume you are interested in studying relationships amongst three variables X

Y and Z Your first hypothesis posits Z as a mediator of the relationship between

X and Y Your second hypothesis posits Z as a moderator of the relationship

between X and Y Draw the figures depicting each of these hypothesized

relationships and describe (make sure to write regression equations) how you

would test each of these hypotheses

D5 The amount of time and effort that the average person pours into his or her job

has risen steadily over the last two decades such that the average person now is

working one more week per year than the average person in the mid 1970s This

steady increase has fueled research interest in the construct of Job Burnout (JB)

which is an emotional and physical exhaustion associated with ones work

Organizational Stress (OS) the perception of greater demands than available

resources in ones work environment is thought to be one of the chief

antecedents of JB However time spent under OS is thought to make its affects

on JB accumulate or worsen Work Role Withdrawal (WRW) the avoidance of

responsibilities or the deliberate reduction of inputs to ones job is believed to be

the chief consequence of JB

a Formally state these ideas as verbal hypotheses as equations and as a complex

directed graph (circle amp arrow diagram)

b Conceptually what is a mediator Define the term and identify the mediator in

your hypotheses in (a) What elements or steps in a study design (not statistics)

that would be helpful for detecting the mediating relationship that was proposed

c What is a moderator What are some of the other names for it within various

subdomains of management Further demonstrate your understanding of

moderator variables by picking which variable in (a) is the moderator Show the

moderating relationship in an X-Y (or multi-layered) chart Again what elements

or steps in a study design (not statistics) would be helpful for detecting the

moderating effect implied above

E) Multiple Comparisons

E1 In your role as professional colleague you are called upon to advise another doctoral

student about the appropriate use of multiple comparisons and post hoc tests in

analysis of variance Your student colleague already has been introduced to ANOVA

and multiple regression Specifically you are to write out a description of how to test

for differences among the different levels or cells of a categorical variable In your

description you should discuss among other things

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

a the statistical and philosophical issues raised by post hoc analysis and data

snooping

b the different multiple comparison methods commonly used by researchers to test

for differences among cell means and

c the advantages and disadvantages of each method

Note Your written description will be evaluated based on your demonstrated

competence with the statistical method not on the pedagogy of your explanation

F) Applied Data Analysis

16) Q Your departmental chair is considering a new textbook and has asked for your

recommendations You conduct a well-designed controlled study (details are unimportant) and discover students who use the new book perform 10 better on a standardized subject

matter exam than those using the old book (p = 027) Both books are readily availabe their prices are almost identical and both are equally acceptable to the instructors What is your recommendation to the chair

F1 Attached is the beginning of an article Your task is to write the results section by

1 making up and reporting plausible descriptive statistics that will help the

reader understand the results of the statistical tests you report and the nature of your

sample (Hint this can be presented in the form of a table)

2 describing and defending your choice of appropriate statistical tests of the

hypotheses and the steps that you would go through in carrying out these tests

3 making up and reporting results of these tests that are consistent with your

descriptive statistics and with any information in the rest of the article about the

results found (Hint this can be presented in the form of a table)

4 describing how well these results provide support for the hypotheses You

may want to note any relevant threats to validity here

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Be sure to state any assumptions you make

F2 Assume that the Personnel Manager of a local firm Ms B dFlandres hired you to

analyze some data that her assistants had collected over the past several years The

data consist of (a) job performance ratings that were made in June of 1990 and (b)

scores on 25 predictor variables (eg measures of education job experience job

tenure job aptitude personality) that were measured when the individuals were hired

(between April 1987 and May 1988) She had data from only the 110 subjects (60

males and 50 females) who were hired during the period and were still employed by

the firm One objective of your analysis (analyses) is to determine how well the

performance criterion can be predicted by the predictors What strategy would you

use in analyzing the data Describeexplain (b) Is sample size an issue for your

analyses Explain If it is what will you do to deal with problems stemming from

sample size Assume that no additional data can be collected at this time (c) Some

preliminary analyses that you do reveal that five of the predictor variables correlate

(zero-order basis) about 75 with one another Will this influence the analyses that

you use Will it influence the validity of conclusions derived from your proposed

analyses (d) Ms dFlandres is interested in knowing if the predictor variables are

equally effective in predicting the job success of males and females What analysis

or analyses would you do to deal with this issue

F3 Assume that you have been provided with data on 12 conceptually distinct variables

that are moderately correlated with one another at the empirical level The data are

from students in four different college majors ie astronomy (A) biology (B)

criminology (C) and demography (D) The 12 measured variables fall into the

general categories of mental health (tension emotional withdrawal hallucinations

guilt feelings blunted affect and suspiciousness) and job satisfaction (work itself

supervision co-workers pay promotion prospects and working conditions) Your

task is to determine if the individuals in the four college majors differ from one

another on the 12 measured variables

(a) What statistical technique(s) would you use to test for group differences

Explain Be specific about the procedures(s) that you would use

(b) What statistical estimate would you use to estimate the extent to which the

groups differ from one another Be specific

F4 The following statistical table published in a recent issue of Work amp Occupations

was taken from an article discussing variables influencing resistance and citizenship

behaviors in the workplace Resistance behaviors include playing dumb withholding

effort work avoidance subverting particular managers and sabotage Citizenship

behaviors include cooperation engaging in peer training use of insider knowledge to

facilitate production commitment to organizational goals giving extra effort freely

Both measures are global constructs aggregated to the interval level and

standardized

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

The table reports the results of a series of ordinary least squares regression equations

with standardized coefficients Explain the meaning of the statistics presented What

do these statistics tell us generally about the relationship between resistance

behaviors and anomie workplace control strategies and workplace characteristics

What do they tell us generally about the relationship between citizenship behavior

and anomie workplace control strategies and workplace characteristics Discuss the

relative advantagesdisadvantages of using standardized regression coefficients in the

data presentation The measurement levels for independent variables are as follows

Anomie Index -standardized interval level measure tapping multiple dimensions

The concept is used by the author to refer to norm violations that workers perceive

Work settings scoring high on the anomie index variable would be ones where

managements act capriciously and arbitrarily in dealing with workers and would not

tend sufficiently to production duties--ie planning providing supplies etc

Forms of Workplace Control -all are dichotomous (coded 1=present 0=absent)

with some overlap within cases Bilateral control refers to cases where management

allows subordinates input into decision making

Job Characteristics -skill and autonomy variables are interval level measures The

union presence variable is dichotomous (1=union 0=no union)

F5 Researchers collected data from a sample of 350 companies on the following

variables

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options low-cost provider or

differentiator)

HR Strategy (a nominal variable with 2 response categories High commitment or

low commitment)

Return on Equity (ROE)

Their proposition is that a firms HR strategy should match up with their business

strategy Specifically they believe that investing in high commitment workforce

practices is needed when a firm is trying to deploy a differentiation business strategy

Furthermore they believe that such a matching is especially important as firms grow

larger

Develop specific statistical hypotheses from these propositions using the variables

described above Then construct a regression model of the form Y = B1 X1 + B2 X2

+ that would test these hypotheses Explain what statistical test you would

examine to reach a conclusion about each hypothesis

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

G) UnivariateBivariate Statistical Procedures

G1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for undergraduate organizational studies students on basic statistical analysis

procedures Your students already have had an elementary course in statistics (eg

probability the normal binomial and t distributions significance testing the logic

of hypothesis testing etc) but have not studied basic data analysis procedures

Outline a tutorial for your students thoroughly describing and explaining one of the

following statistical analysis techniques One-way ANOVA Pearson product-

moment correlation or the chi-square measure of association (Your tutorial can be

conceptual it need not present any mathematical formulas unless you wish to) In

your tutorial be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

H) Multivariate Statistical Analysis

H1 In your role as Teacher of Organizational Studies you are asked to prepare a tutorial

for graduate organizational studies students on multivariate statistical analysis

procedures Your students already have had the basic course in bivariate statistics

(eg t-test F-test ANOVA correlation etc) and are beginning their study of

multivariate data analysis techniques Outline a tutorial for your graduate students

thoroughly describing and explaining one of the following multivariate statistical

analysis techniques Multiple regression factor analysis discriminant analysis

canonical analysis cluster analysis or MANOVA (Your tutorial can be conceptual

it need not present any mathematical formulas unless you wish to) In your tutorial

be sure to discuss (among other issues)

1 The type of output that results from applying the technique How to interpret

results from the technique

2 The assumptions underlying the technique and the consequences of failures in the

assumptions

(Note Your tutorial will be evaluated on your demonstrated competence with the

statistical method not on the pedagogy of the tutorial)

I) ANOVA

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

I1 ANOVA Assume that you have conducted a laboratory experiment in which there

were 3 levels of Factor A (a1 a2 a3) and two levels of Factor B (b1 b2) Assume also

that there were 20 subjects in each cell

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this

design Make certain that you specify all effects and the appropriate degrees of

freedom for each effect If you use any abbreviations in your table make certain that

you specify their meaning

(b) Assume that an initial ANOVA analysis found no evidence of an A X B

interaction What if anything would you do with the sum of squares for the

interaction effect

I2 A study examined the effects of gender and pre-training motivation on the transfer of

training in terms of improved performance for 123 new industrial managers The

training was on supervisory skills and the dependent variable was based on the

appraisal of the supervisors performance before the training and three months after

the training The performance appraisal data were collected for research purposes

only and resulted in a total transfer score from 1 to 25 with 1 being low Pre-training

motivation was based on a 12 item Likert scale with an alpha of 86 Two groups

were identified for the pre-training motivation variable - the bottom 13 (low) and the

top 13 (high) Table 1 contains the ANOVA results and Table 2 contains the cell

means

a What experimental design was used in this study

b What assumptions are made in the ANOVA model

c How would you interpret the results in Tables 1 and 2

d What additional computation can be done to evaluate the importance of the

different effects in Table 1

e Using the results in Table 2 show the G x M effect in a graph (It is OK to draw

this graph on a separate sheet and turn it in with your exam)

f Explain what type I and type II error mean in the context of this study

Table 1 ANOVA Results for the Transfer of Training Study

Source of Variation SS df MS F

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Gender (G) 24974263 1 24974263 235

Pre-training motivation (M) 42695979 1 42695979 402

G x M 125777702 1 125777702 1184

Residual 1264266700 119 10624090

Totals 1457714644 122

_________________________________________________________________

plt 05 plt 001

Table 2 Cells Means

Pre-training Motivation

Gender Low High

Male 135 216

Female 157 183

J) Meta-analysis

J1 There exist two categories of literature review narrative and quantitative Each

category as typically implemented has certain advantages and disadvantages

Describe the advantages and disadvantages associated with each approach taking

particular care to enumerate the disadvantages associated with quantitative review

Then offer two ways that meta-analysis might be combined with other analytical

techniques to move beyond the focus of bivariate relationships that is typical of meta-

analysis

J2 Schmidt has argued that primary research does not allow one to discover general

scientific laws but that meta-analytic research does What are the arguments in favor

of or against this position In what areas of research (if any) has Schmidts

prediction come closest to coming true

J3 The so-called (by Hunter Schmidt and Jackson 1983) state of the art meta-

analysis method attempts to account for various sources of error that might produce

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

apparent variation in effect sizes across studies (a) Discuss the three most important

sources of error that meta-analysis addresses (b) Discuss the potential contribution

that a meta-analysis can make to theory building in the organizational sciences (c)

Describe the circumstances that you would consider when deciding whether to

perform a meta-analysis

J4 Meta-analysis has been hailed as a miracle cure and criticized as smoke and

mirrors Explain both sides of the debate and then lay out the role of meta-analysis

as you see it within the time span of your career as an academic To do this use

examples of already written (or yet to be written) articles as models

J5 A frequent criticism of non-quantitative research summaries (literature reviews) is

that researchers do not carefully examine how particular research designs might have

affected the statistical outcomes of the studies included in the literature review Meta-

analysis however allows one to statistically examine such effects Assume you are

conducting a meta-analysis of the relationship between leader behaviors and group

productivity Briefly describe the steps you would take in conducting a meta-

analysis What aspects of the original research designs would you examine in your

meta-analysis What steps would you take in gathering and coding data to enhance

the credibility of your analysis

K) MTMM

K1 One method often used by social scientists to demonstrate the construct validity of

measures is the multi-trait multi-method matrix developed by Campbell and

Fiske (1959) Briefly describe how this matrix is obtained or constructed by a

researcher and then describe how each of the elements of the matrix is used to

demonstrate the reliability andor construct validity of the measure(s)

IC

K2 Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of

the criteria for evaluating the validity of a set of measurement procedures

Interpret the following MTMM in terms of what it might infer about the validity

of the different measurement procedures You may impose any assumptions on

the data you prefer For example you might want to discuss the matrix under the

assumption that traits A B amp C are mutually independent Please justify and

make explicit any assumptions you make

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3

A1 (89)

Method I B1 51 (89)

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

C1 38 37 (76)

A2 57 22 09 (93)

Method II B2 22 57 10 68 (94)

C2 11 11 46 59 58 (84)

A3 56 22 11 67 42 33 (94)

Method III B3 23 58 12 43 66 34 67 (92)

C3 11 11 45 34 32 58 58 60 (85)

(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1 Categorical variables can be analyzed in regression through the use of design

variables Three most common design variables are dummy codes effect codes

and contrast codes 1) Define compare and contrast these approaches to coding

2) offer and describe substantive examples to which each approach is uniquely

suited and 3) describe how interactions among categorical variables are examined

in regression

L2 Categorical variables can be analyzed in regression using design variables The

three most common design variables are dummy codes effect codes and contrast

codes Define compare and contrast these approaches to coding and describe

what information is provided by the beta weights in each approach

L3 Organizational researchers often encounter the problem of multicollinearity in

their data analyses Answer the following questions about this topic

(a) What is multicollinearity

(b) What are the sources of multicollinearity

(c) What are the effects of multicollinearity on research

(d) What are some diagnostics that researchers can use to detect multicollinearity

What are some methods for dealing with multicollinearity Discuss the adequacies

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

of each method described

L4 Various procedures can be used in carrying out a multiple regression analysis

Describe the reasons or conditions under which each of the five procedures listed

below would be an appropriate analytic strategy In addition discuss how you

would interpret the results yielded by the different procedures

a Simultaneous multiple regression

b Hierarchical multiple regression

c Stepwise multiple regression - forward

d Stepwise multiple regression - backwards

e Moderated multiple regression

IC

L5 Multiple regression is probably the statistical analysis procedure most commonly

used in social science Assume that you have a junior colleague who has had a

basic course in bivariate statistics but never has studied multiple regression Write

a brief tutorial for your colleague describing and explaining multiple regression

(Your tutorial can be conceptual it need not present any mathematical formulas

unless you wish to)

L6 Assume the following regression model Y = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y values are to be estimated using standard score (Z) data

(a) What does b2 represent Explain in a clear and concise manner

(b) What does b3 represent Explain in a clear and concise manner

(c) What are the two most important determinants of the statistical significance of

any of the i values Specify and indicate how each is related to statistical

significance

L7 Multiple regression is probably the single most frequently used multivariate data

analysis procedure in social science It is imperative to understand the methods

assumptions interpretation and strengthsweaknesses of that oft-used data

analysis methodology

Define and discuss important features or issues concerning the following aspects

of multiple regression

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Outputs of Regression Analysis

F-test of significance and its associated degrees of freedom

Regression coefficient (Beta weight) standardized andor non-standardized

R2 The proportion of the total sum of squares that is accounted for by the

predictors (linear and with non-linear terms)

Shrunken or adjusted R2 (a downward adjustment in the population estimate

of R2 based on its estimation in a sample of size N)

Outliers

Assumptions and Limitations of the Method

Assumptions (describediscuss the assumptions underlying multiple regression)

(Normality and equality of variance independence of the observations linearity

interval-level measurement of all variables)

Multi-colinearity

The least-squares criterion

Methods of Regression

Step-wise regression

Non-linear regression

Moderated regression

M) General Linear Model

M1 The general linear model underlies most multivariate statistical methods Describe

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

and discuss the assumptions of the general linear model Multiple regression

ANOVA and principle components (factor) analysis are based on the general linear

model For each of those three statistical procedures describe and discuss the

following

a The type of data necessary for using the technique appropriately

b The type of output that results from applying the technique

c The relative advantages and disadvantages of each technique

d The nature of the dependent and independent variable in each technique

M2 Regression and ANOVA are based on the general linear model Describe the

general linear model and discuss its assumptions For both regression and ANOVA

discuss the following

a The assumptions of the techniques and the type of data necessary for using the

technique properly

b The output that results from applying the technique How to interpret results

c The relative advantages and disadvantages of each technique

N) Clustering

N1 A recurring measurement issue is the assignment of objects to groups

(categoriesdimensions) based upon their similarities Describe analytic techniques

for performing this analysis How could you assess the similarity of the objects

How could you assign similar objects to groups (categoriesdimensions) When

answering this question be sure to address at least the following issues

a The type of data necessary for the use of each technique

b The type of output that results from the application of the technique

c The criteria (and procedure) for determining the number of factors or groups

d The relative advantages and disadvantages of the technique

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1 Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis

(CFA) Give an example for the appropriate use of each of the above analyses

What is the role of theory in the two factor analyses

IC

O2 Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory TH

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

Factor Analysis (CFA) Put more emphasis on their theoretical basis

assumptions usage and limitations

O3 How do exploratory and confirmatory factor analysis differ What are the

advantages and disadvantages of each

O4 Describe the differences between exploratory factor analysis (EFA) and

confirmatory factor analysis (CFA) When would you use one verses another

Are there ways to use EFA in a confirmatory way Can you use CFA in an

exploratory way Also describe the difference in principal components analysis

(PCA) and EFA

O5 There has been a remarkable convergence of evidence in the personality

(individual differences) literature in the last ten years toward the Big Five

theory of personality structure So if a management researcher is interested in

the enduring traits of top managers job candidates boundary role spanners or

any set of individuals it is now requisite to measure one or more of the Big

Five characteristics Openness to Experience (O) Conscientiousness (C)

Extraversion (E) AgreeablenessConformity (A) and NeuroticismNegative

Affectivity (N) Unfortunately almost all of the well-established measures of

these personality dimensions are copyrighted and cost $1 or more per subject to

use in research Hence there is a clear need for someone to develop a public

domain set of measures of the Big Five

Suppose you were on a research team to develop that set of measures and youre

aiming for having 100 total items with 20 in each dimensional subscale Suppose

also that you collect data using these new items from 350 undergraduates who

also give your scale to their best friend so that you can have both self- and

social-reports of the undergraduates personalities (350 people being rated with

100 self-report items and another 100 social report items = 200 items total) How

would you use exploratory and confirmatory factor analysis (EFA and CFA) to

examine and help to validate your public domain measures Specifically

a -d What steps would you go through and what choices would you make in

performing an EFA on this data set In your answer make some justified guess

at the factor structure you expect to find

Armed with your EFA results how would you use (sketch out a plan without

getting into great detail regarding degrees of freedom and so on) CFA to provide

a more formal test of the fidelity of your measure In your answer distinguish

the important differences in purposes assumptions and mechanics of the two

types of techniques How does hypothesis testing differ in this context relative to

others such as regression

O6 The basic factor analysis model can be represented by the matrix algebraic

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

equation A B = C

(1) What do each of these matrices represent Be specific as to the type of

information in each matrix and its purpose

(2) For what purpose do researchers use factor analysis Explain In the process

provide a very brief description of a study for which factor analysis would be a

useful data analytic technique

O7 Factor analysis is an important statistical technique in organizational research

Describe factor analysis as an analytical tool including the following issues

a the reasons for using factor analysis

b the type of data necessary for using the technique appropriately

c the type of quantitative output that results from applying the technique

d how a researcher would use the output in his or her research

e how confirmatory factor analysis is different from exploratory

f the important issues and problems in using the technique

P) Structural Equation Models

P1 Structural equation modeling (SEM) has become one of the more popular data

analytic techniques in the social sciences Like any analytic technique it has its

advantages and disadvantages 1) Explain the advantage of SEM over regression-

based path analysis associated with measurement error 2) explain the notion of

maximum likelihood estimation broadly defined 3) define compare and

contrast any three of the fit indices offered by LISREL and 4) discuss

unmeasured variables problem ala James (JAP 1980) and how it affects

parameter estimation

P2 There now exist dozens of model fit for SEM Tanaka (in Bollen amp Long 1993

Testing structure Equation Models) offered six dimensions on which fit indices

vary The four dimensions with which you are concerned are Simplicity vs

Complexity Normed vs Nonnormed Absolute vs Relative and Sample size

independent vs Size dependent Your tasks are to explain these four distinctions

and to give examples of fit indices that represent each category (eg Normed fit

index and nonnormed fit index) explaining briefly why each index is

representative of the category

P3 An important consideration in structural equation modeling is whether manifest

variables should consist of individual items or item parcels (ie sums or averages

of individual items) For instance a construct measured with 10 items may be

specified by treating each item as a separate manifest variable or by averaging

TH

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

pairs of items thereby yielding five manifest variables The use of individual

items versus item parcels has generated considerable controversy (eg Bandalos

2002 Bandalos amp Finney 2001 Landis Beal amp Tesluk 2000 Marsh Hau

Balla amp Grayson 1998) Some researchers advocate the use of item parcels

whereas other research argue in favor of using individual items Discuss the

relative merits of using individual items and item parcels in structural equation

modeling Summarize your discussion by presenting guidelines that stipulate

when and under what conditions organizational researchers should use individual

items versus item parcels

Bandalos D L (2002) The effects of item parceling on goodness-of-fit and

parameter estimate bias in structural equation modeling Structural Equation

Modeling 9 78-102

Bandalos D L amp Finney S J (2001) Item parceling issues in structural

equation modeling In G A Marcoulides amp R E Schumacker (Eds) New

developments and techniques in structural equation modeling (pp 269-296)

Hillsdale NJ Erlbaum

Landis R S Beal D J amp Tesluk P E (2000) A comparison of approaches to

forming composite measures in structural equation models Organizational

Research Methods 3 186-207

Marsh H W Hau K T Balla J R amp Grayson D (1998) Is more ever too

much The number of indicators per factor in confirmatory factor analysis

Multivariate Behavioral Research 33 181-220

P4 Some researchers argue that Structural Equation Modeling (SEM eg LISREL)

is a very powerful data analytic technique which has not been applied enough in

substantive research while others would argue that its benefits have been greatly

exaggerated and that it has been overapplied Consider these perspectives by

discussing the various strengths and weaknesses of SEM In your discussion you

should address among other things how SEM compares to other data analytic

techniques What do you think about the underapplicationoverapplication of

SEM

P5 Structural equation modeling has gone from an obscure and complicated

statistical tool to one that is commonly demanded by reviewers and used by

relative novices As SEM becomes more user friendly a number of statistical

experts have voiced concerns about the wisdom of making programs like LISREL

and EQS accessible to the masses (with simplified programming languages and

graphic user interfaces)

Please comment on these restrictions versus access argument Which side do you

take

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam

What are the strengths of SEM Give some examples

What are the weaknesses of SEM Give some examples

MANY THANKS TO OUR CONTRIBUTORS

Arthur G Bedeian Kristin Byron Wendy Casper Gilad Chan Gordon Cheung Janet Dukerich Jeff Edwards Daniel

Ganster Mark Gavin Jodi Goodman Dave Harrison Mickey Kavanagh Adam Meade Dan Newman Catherine

Schwoerer Byran Schaffer Tom Taber Jeffrey Vancouver James Vanscotter Susan Winter

1 1 IC = In Class Exam TH = Take Home Exam