1

Group-Level Measurement

Katherine KleinUniversity of Pennsylvaniakleink@wharton.upenn.edu

CARMA PresentationFebruary 2007

2

Why Group-Level Measurement?

• Burgeoning of multilevel theory and research in last 25 years

• Great progress in conceptualizing and measuring group-level constructs

– Especially shared constructs

• Continuing challenges and opportunities– Especially regarding configural constructs

3

A Few Terms and Assumptions

• I’ll refer to groups but much or all of what I say will apply as well to organizations, departments, stores, etc.

• I’ll focus on the creation and use of original survey measures to assess group constructs.

• I’ll address statistical issues in passing only.– But see past CARMA presenters including James

LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others

4

Roadmap Fundamentals: Theory First

Construct Types: Global, Shared, and Configural Constructs

Practicalities and Technicalities Survey Wording Sampling Qualitative Groundwork Single-source Bias Justifying Aggregation

Opportunities and Challenges The Configuration of Diversity Social Network Analysis

5

Fundamentals: Theory First

• Constructs are our building blocks in developing and in testing theory.

• High quality measures are construct valid.

• The development of construct valid measures thus begins with careful construct definition.

• Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): – Global, shared, or configural.

6

Global Constructs

• Relatively objective, easily observable, descriptive group characteristics.

• Originate and are manifest at the group level.

• Examples: – Group function, size, or location.

• No meaningful within-group variability.

• Measurement is generally straightforward.

7

Shared Constructs

• Group characteristics that are common to group members

• Originate in group members’ attitudes, perceptions, cognitions, or behaviors– Which converge as a function of attraction, selection,

socialization, leadership, shared experience, and interaction.

• Within-group variability predicted to be low.• Examples:

– Group climate, norms, leader style.

• Measurement challenges are well understood.

8

Configural Group-Level Constructs

• Group characteristics that describe the array, pattern, dispersion, or variability within a group.

• Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes)– But no assumption or prediction of convergence.

• Examples: – Rates, diversity, fault-lines, social networks, team mental

models, team star or weakest member.

• Measurement challenges are less well understood.

9

A Related Framework: Chan’s (1988) Composition Typology

• Shared Constructs– Direct consensus models (e.g., group norms) – Referent shift models (e.g., team efficacy)

• Configural Constructs– Dispersion model (e.g., climate strength)– Additive models (e.g., mean group member IQ)

• Multilevel, Homologous Models– Process model (e.g., efficacy-performance

relationship)

10

Construct Definition Complexities:An Example: Shared Leadership

• Shared leadership– “A dynamic, interactive influence process among

individuals in work groups in which the objective is to lead one another to the achievement of group goals… [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence”

– Conger & Pearce, 2003, p. 286

• Is this a shared construct, or a configural construct, or … ?

11

Construct Definition Complexities:An Example: Shared Leadership

• Well, how would you measure it?

– Shared team leadership as a shared construct• “Team members share in the leadership of this team.”• “Many team members provide guidance and direction for

other team members.”

– Shared team leadership as a configural construct (network density):

• “To what extent do you consider _____ an informal leader of the team?”

12

Construct Definition Complexities:An Example: Shared Leadership

• Calling it a “referent shift” construct is not the answer.– Referent shift is a measurement strategy, not a

construct type

• Shifting the referent in an unthinking manner can be quite problematic:– The members of my team…

• “Express confidence that we will achieve our goals” • “Will recommend that I am compensated more if I perform

well” • “Are friendly and approachable” • “Rule with an iron hand”

13

A Quick Recap

• Theory first: Define and explain the nature of your group-level constructs.

– Is it a clearly objective description of the group? • If yes, a global construct.

– Do you expect within-group agreement? • If yes, a shared construct.

– Does it describe the group in terms of the pattern or array of group members on a common attribute?

• If yes, a configural construct.

14

Now What?

• Having defined your constructs, the goal is to create measures that:– Are construct valid– Show homogeneity within (shared constructs)– Show variability between (all group-level constructs)

• Practicalities and technicalities– Survey wording– Sampling– Qualitative groundwork– Minimizing single-source bias– Testing for aggregation

15

Survey Wording:Global Constructs

• Draw attention to objective descriptions of each group.

• Gather data from experts and observers (SMEs) who can provide valid information about the groups in question.

• No need to gather data from individual respondents within groups

• Use language that fits your sample.

16

Survey Wording: Shared Constructs

• Draw attention to shared group characteristics

• Use a group referent rather than individual referent to enhance: – Within group agreement– Between group variability– Predictive validity

• Gather data from individual respondents so within-group agreement can be assessed.

• Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.

17

Survey Wording:Configural Constructs

• Draw attention to individual group member characteristics by using an individual referent.

• Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups.

• The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.

18

Sampling

• Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest

• Whether they are global, shared, or configural.

• Statistical power reflects both: – Group sample size (n of groups)– Within-group sample size

• When group size is large (number of respondents per group), measures of shared constructs are more reliable.

• More research needed on power in multilevel analyses.

19

Qualitative Groundwork

• The survey wording and sampling guidelines seem fairly obvious and easy, but …

• Check your assumptions in the field prior to survey data collection.– Are you measuring the right “groups”?

• Example: Grocery stores or departments?– Is there meaningful between-group variability?

• Example: Fast food chain– Are you measuring the right variables, and not too

many of them?• Beware the blob.

20

Single-Source Bias

• Group-level correlations between measures of shared group constructs may be disturbingly high.– Examples:

• Transformational and transactional leadership• Task, emotional, and procedural conflict

• Aggregation does not “average away” response biases.• Rather, group members may share response biases

– Halo, logical consistency, social desirability

• Response bias may be particularly influential when respondents must make subtle distinctions among constructs.

21

Single-Source Bias:Beating the Blob

• Survey measures– Choose and measure truly distinct constructs– Use different survey response formats

• Survey design– Keep survey items measuring distinct

constructs separate.• Help respondents recognize the distinction

between leadership types, or conflict types, for example.

22

Single-Source Bias:Beating the Blob

• Survey analysis– Randomly split the within-group sample of

respondents during data analysis.• All receive the same survey, but half provide IV and the other

half provide the DV for analyses

• Survey administration– Randomly split the within-group sample of

respondents during data administration. • Respondents receive distinctive surveys. Half receive the IV

survey and the other half receive the DV survey.

23

A Quick Recap

• Having– Defined our constructs– Written our survey items– Conducted qualitative groundwork– Sampled appropriately– Taken steps to reduce single source bias

• We’re almost ready for hypothesis testing

• But first: We need to justify aggregation

24

Justifying Aggregation

• Why is this essential?– In the case of shared constructs, our very construct

definitions rest on assumptions regarding within- and between-group variability.

– If our assumptions are wrong, our construct “theories,” our measures, and/or our sample are flawed and so are our conclusions.

• So, test both:– Within group agreement

• The construct is supposed to be shared, but is it really?– Between group variability (reliability)

• Groups are expected to differ significantly, but do they really?

25

Justifying Aggregation: rwg(j)

• Developed by James. Demaree, & Wolf (1984)• Assesses agreement in one group at a time.• Compares actual to expected variance.• Answers the question:

– How much do members of each group agree in their responses to this item (or this scale)?

• Highly negatively correlated with the within group standard deviation

• Valid values range from 0 to 1• Rule of thumb: rwg(j) of .70 or higher is

acceptable

26

Justifying Aggregation: rwg

• Common to report average or median rwg(j) for each group for each variable:– If rwg(j) is below .70 for one or more groups, check:

• Does the group have low rwg(j) values on several variables?

• Do many groups have low rwg(j) values on this variable?

• Remember: rwg(j) indicates within-group agreement, not between-group variability.

• Beware: When variance in a group exceeds expected variance, out of range rwg(j) result.

27

Justifying Aggregation: 2

• Assesses between-group variance relative to total variance, across the entire sample.

• Based on a one-way ANOVA• Answers the question:

– To what extent is variability in the measure predictable from group membership?

• The F-test provides a test of significance– The larger the sample of individuals, the more likely

eta2 is to be significant.• Beware: 2 may be inflated when group sizes

are small (under 25 individuals per group)– But, this is an easy way to begin tests of aggregation

28

Justifying Aggregation: ICC(1)

• Assesses between-group variance relative to total variance

• Based on a one-way ANOVA• Answers the question:

– To what extent is variability in the measure predictable from group membership?

• The F-test provides a test of significance• Based on 2 but controls for the number of

predictors relative to the total sample size, so ICC(1) is not biased by group size.

29

Justifying Aggregation: ICC(2)

• Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size.

• Answers the question: – How reliable are between-group differences on the

measure?

• Reflects ICC(1) and within-group sample size– Example: If ICC(1) = .20 and:

• Mean group size is 5, expected ICC(2) = .56• Mean group size is 20, expected ICC(2) = .71

30

Justifying Aggregation: An Example

2 ICC(1) ICC(2) Average rwg(j)

Financial Resource Availability

.28*** .21 .75 .65

Mgt. Support .31*** .19 .61 .78

Policies and Practices

.33*** .22 .74 .79

Implement. Climate

.23*** .15 .64 .81

31

A Quick Recap

• The hope is that we have successfully:– Defined our constructs.– Written our survey items.– Conducted qualitative groundwork.– Collected data from a large sample of groups.– Taken steps to reduce single source bias.– Justified aggregation.– And moved on to test our hypotheses.

• So, what remains?

32

Opportunities and Challenges:The Configuration of Diversity

• Configural constructs describe the array, pattern, dispersion, or variability within a group.– The easy example is diversity

• Demographic diversity• Climate strength

• But even the easy example isn’t so easy: What is the definition of diversity? And how should it be measured?

33

The Configuration of Diversity

• A starting definition of diversity:– The distribution of differences among the members of

a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay.

• Okay, but what’s maximum diversity? – Which team has maximum age diversity?

• 20, 20, 20, 70, 70, 70• 20, 30, 40, 50, 60, 70• 20, 20, 20, 20, 20, 70• 20, 70, 70, 70, 70, 70

34

The Configuration of Diversity

• Diversity isn’t one thing.

• It’s three things: Separation, Variety, or Disparity

• The three types differ in: – Meaning or substance – Pattern or shape – Likely consequences– Appropriate operationalization

• Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions

35

The Configuration of DiversityExample: Three Research Teams

• Team S – Members differ in their view of qualitative research.

• Half of the team members respect it, half don’t.

• Team V– Members differ in their discipline.

• 1 psychologist, 1 sociologist, 1 anthropologist, etc.

• Team D– Members differ in their rank

• 1 senior professor, others are incoming graduate students.

36

Diversity as Separation

• Differences in group members’ position, attitude, or opinion along a continuum

• Min: Every member has the same opinion

• Max: Two polarized extreme factions

• Theory: Similarity-attraction

• Operationalization: Standard deviation

37

Diversity as Variety

• Differences in kind or category

• Min: Every member is the same type

• Max: Each group member is a different type

• Theory: Requisite variety, cognitive resource heterogeneity

• Operationalization: Blau’s index of categorical differences

38

Diversity as Disparity

• Differences in concentration or proportion of valued assets or resources

• Min: Every member has an equal portion of the resource

• Max: One member is “rich” and all others are “impoverished”

• Note: Disparity is asymmetric

• Theory: Inequality, relative deprivation, tournament compensation

• Operationalization: Coefficient of variation (SD/Mean)

39

The Configuration of Diversity:A Recap

• Theory first– Separation is about position, attitude, or opinion

– At maximum: Polarized factions

– Variety is about knowledge or information.– At maximum: One of a kind

– Disparity is about resources or power. – At maximum: One towers over others

• Operationalize accordingly– The coefficient of variation is not a default or catch-all

40

Opportunities and Challenges: Social Network Analysis

• Multilevel analysis and social network analysis have developed along separate paths.

• Rich opportunities for cross-fertilization.

• Social network analysis provides a means to conceptualize and operationalize configural constructs.– Illuminating the pattern or array of interpersonal ties

within a group

41

Opportunities and Challenges: Social Network Analysis

• Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups.

• Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict

• But we know little about the configuration of interpersonal ties – the structures – that underlie our shared constructs and measures.

42

An Example: Social Network Analysis and Shared Team Conflict

• When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team?

• As a starting point:– How dense are positive (advice) ties? – How dense are negative (difficulty) ties?

43

An Example: Social Network Analysis and Shared Team Conflict

Task and emotional conflict: The blob r = .83

Advice density and negative tie density: More weakly correlated r = -.36

Task conflict (mean task and emotional conflict), advice density, and negative tie density Team Conflict and Advice Density: r = -.47 Team Conflict and Difficulty Density r = .40

44

Negative Tiesin a Low Conflict Team

45

Negative Ties in a High Conflict Team

46

Advice Ties in a High Conflict Team

47

Advice Ties in a Low Conflict Team

48

Social Network Analysis:A Recap

• Social network analysis illuminates the configuration of interpersonal ties in groups.– What network structures underlie our shared

constructs and measures? – Do network measures provide incremental validity?

• Not just density, but centralization, cliques, and more.

• What explains between-group differences in network structures?

49

In Conclusion

• Theory first. Define your constructs. – Are they global, shared, or configural?

• Measure constructs and collect data with care– Match item wording to the construct – Conduct qualitative groundwork– Sample appropriately– Take steps to reduce single source bias– Test for aggregation

• Studying configural constructs remains a challenge and an opportunity– Conceptualizing and measuring diversity– Integrating social network analysis within our arsenal

50

Some Helpful References

1. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.

2. Borgatti, S. P. (2003). The network paradigm in organizational research: A review and typology. Journal of Management, 29, 991-1013.

3. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.

4. Harrison, D. A. & Klein, K. J. (2007). What’s the difference? Diversity as separation, variety, or disparity in organizations. Academy of Management Review.

5. Harrison, D. A. & McLaughlin, M. E. (1996). Structural properties and psychometric qualities of organizational self-reports: Field tests of connections predicted by cognitive theory. Journal of Management, 22, 313-338.

6. James, Demaree, & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.

51

Some Helpful References7. Klein, K. J., Conn, A. B., Smith, B., & Sorra, J. S. (2001). Is everyone in

agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.

8. Klein, K. J., Conn, A. B. & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86, 3-16.

9. Kozlowski, S. W. J. & Klein, K. J. (2000). A multilevel approach to theory and research in organizations. In Klein, K. J. & Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3-90). San Francisco: Jossey-Bass.

10. Morgeson, F. P. & Hofmann, D. A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24, 249-265.

11. Ostroff, C., Kinicki, A. J., & Clark, M. A. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfactoin relationships. Journal of Applied Psychology, 87, 355-368.