volume 13 number 2 construct validation using 2010 sage

Construct Validation UsingComputer-Aided Text Analysis(CATA)

An Illustration Using Entrepreneurial Orientation

Jeremy C. ShortTexas Tech University

J. Christian BrobergWichita State University

Claudia C. CogliserKeith H. BrighamTexas Tech University

Construct validity continues to pose challenges in the organizational sciences. To capture

difficult-to-measure constructs of interest, researchers have often relied on content analysis.

One content analysis technique, computer-aided text analysis (CATA), is particularly

attractive because of the ability to process large samples with high speeds and reliabilities.

Unfortunately, inconsistent guidance exists to guide researchers through the use of this tool

in a manner compatible with accepted methods used to validate constructs in a rigorous

manner. The authors review research using content analysis to examine the extent to which

such studies integrate methods for assessing content, external, discriminant, and predictive

validity. To provide direction for organizational researchers interested in using CATA to

measure theoretically based constructs relevant to the management field, they suggest a

number of possible procedures to enhance construct validity. They illustrate these

procedures using the construct of entrepreneurial orientation.

Keywords: computer-aided text analysis; construct validation; content analysis; DICTION;

entrepreneurial orientation

Validity refers to evaluating inferences drawn from measures of concepts of interest and

considers the extent to which a measure accurately represents that focal concept

(Cronbach, 1971). The importance of rigor in construct measurement cannot be minimized.

Kerlinger and Lee (2000) argued that the ability to demonstrate appropriate measures of

theoretical notions of interest through construct validity ‘‘is one of the most significant

advances of modern measurement theory and practice’’ (p. 670).

Despite advancements in the organizational sciences vis-a-vis construct validation, reviews

of construct validity in the management field have yielded generally bleak conclusions about

Authors’ Note: The authors would like to thank Tyge Payne for his thoughtful comments on previous drafts of this

manuscript. Please address correspondence to Jeremy C. Short, Area of Management, Texas Tech University,

Lubbock, TX 79408; e-mail: [email protected].

Organizational

Research Methods

Volume 13 Number 2

April 2010 320-347

# 2010 SAGE Publications

10.1177/1094428109335949

http://orm.sagepub.com

hosted at

http://online.sagepub.com

320

its adequacy (e.g., Bagozzi, Yi, & Phillips, 1991; Boyd, Gove, & Hitt, 2005; Scandura &

Williams, 2000). One reason that management scholars use measures that are psychometri-

cally inadequate may be because of the difficulty encountered when implementing survey

methods with samples that typically receive a less-than-ideal response rate in certain business

settings (Baruch, 1999). This is particularly problematic in fields such as entrepreneurship,

where response rates have often been much lower than the management field as a whole

(Bartholomew & Smith, 2006). Likewise, a number of conceptual phenomena of interest

in fields such as strategic management often involve ‘‘unobservables,’’ making rigorous con-

struct validity efforts challenging; for example, the concept of opportunism from transaction

cost theory is inherently unobservable as are the unique resources that could lead to perfor-

mance advantages according to the resource-based view of the firm (Godfrey & Hill, 1995).

To remedy construct measurement problems germane to the management field, research-

ers have often relied on content analysis (Duriau, Reger, & Pfarrer, 2007; Morris, 1994), a

research method that uses a set of procedures to classify or otherwise categorize communi-

cation allowing inferences about context (Krippendorff, 2004; Weber, 1990). Typically

relying on archival narratives to extract criteria of interest to management scholars, content

analysis has been applied to assess elements such as CEO performance comparisons (e.g.,

Short & Palmer, 2003), aspects of organizational sensemaking (e.g., Gioia & Chittipeddi,

1991), and processes such as organizational sensegiving (e.g., Maitlis & Lawrence,

2007). Analysis is generally performed on organizationally produced texts such as CEO

shareholder letters, annual reports, and mission statements (Duriau et al., 2007).

This article is intended to fill a gap in the research methods literature by providing an

enhanced discussion of potential approaches to construct validation, when computer-aided

text analysis (CATA) is the primary tool used to capture theoretically based constructs of

interest for organizational researchers. Examples of software appropriate for CATA include

VBPro, CATPAC, Concordance, DICTION, General Inquirer, LIWC, NVivo, and MECA

(Krippendorff, 2004; Skalski, 2002). Integrating CATA to measure constructs of interest

provides numerous advantages to organizational researchers because CATA has higher

reliability than human coding with lower cost and greater speed (Neuendorf, 2002); yet,

a review of content analysis in organizational studies found that less than 25% of the articles

in major management journals in the past 25 years used CATA in their content analysis pro-

cesses (Duriau et al., 2007). The fact that incorporation of such techniques represents the

exception rather than the norm suggests that construct validation when using content anal-

ysis may be less than ideal and that scholars may not understand how to incorporate such

analyses into their empirical tests. To highlight the potential of CATA for future empirical

efforts, we highlight three contributions to organizational research methods with the goal of

providing a ‘‘user’s guide’’ to CATA, with the potential to offer greater validity than pre-

vious applications of content analysis.

First, we examine the degree to which methods for addressing validity are incorporated

in organizational studies using content analysis. We examine and build on the sample used

in the review of Duriau et al. (2007) organizational studies using content analysis and exam-

ine the content, external, discriminant, and predictive (nomological) validity of such work.

Content analysis experts have suggested that elements of validity such as nomological or

predictive validity are rare in studies using content analysis (McAdams & Zeldow, 1993;

Neuendorf, 2002; Weber, 1990), and such issues may also be problematic in organizational

Short et al. / Construct Validation Using CATA 321

studies using content analysis. Examination of other elements surrounding the use of con-

tent analysis in the organizational literature may also yield insight relevant to construct vali-

dation procedures.

Our second contribution, following in the tradition of other work appearing in Organiza-

tional Research Methods (e.g., Fendt & Sachs, 2008; Karren & Barringer, 2002; Lance,

Butts, & Michels, 2006; Rogelberg & Stanton, 2007), is to provide a ‘‘user’s guide’’ for

future empirical efforts consistent with the goals and values of organizational research. For

example, little guidance exists to aid researchers for developing specialized or custom dic-

tionaries that are needed when using content analysis with individual words as the unit of

analysis. This is somewhat surprising given that a number of predefined dictionary-based

programs have been accepted into the management literature. For example, the predefined

dictionaries in the DICTION software program have been used to examine the language of

charismatic leadership (e.g., Bligh, Kohles, & Meindl, 2004a, 2004b). Short and Palmer

(2008) argued that future efforts could advance management research by supplementing

these predefined dictionaries provided by content analysis software programs with

researcher-specified custom dictionaries. Content analysis experts such as Krippendorff

also noted that ‘‘Most content analyses would benefit from the construction of special-

purpose dictionaries, but developing a dictionary from scratch can be a formidable task.

It is not surprising, therefore, that content analysts usually try to build on available diction-

aries before they attempt to develop their own’’ (2004, p. 287). The suggestion to create

custom dictionaries needed to advance the management field is unlikely to become reality

if guidance is not provided for a thoughtful approach toward building such dictionaries.

At the same time, some advice offered in content analysis texts may not always be com-

pletely compatible with the goals of management researchers using content analytic meth-

ods, who generally incorporate both deductive and inductive approaches when using

content analysis (Doucet & Jehn, 1997; L. Doucet, personal communication, December

14, 2008; B. Kabanoff, personal communication, December 14, 2008; T. Pollock, personal

communication, December 14, 2008). Indeed, such approaches, common in organizational

applications, can be contrasted with content analysis texts such as Krippendorff (2004) who

suggested, ‘‘Deductive and inductive inferences are not central to content analysis’’ (p. 36).

Thus, the specific guidance offered in content analysis texts may result in inconsistent

applications for organizational scholars. For example, when presenting a flowchart for a

typical content analysis process, Neuendorf (2002) suggested:

You may use standard dictionaries (e.g., those in Hart’s program DICTION) or originally cre-

ated dictionaries. When creating original dictionaries, be sure to first generate a frequency list

from your text sample, and examine for key words and phrases. (p. 50)

The inductive suggestion to first examine a frequency list based on a particular text sam-

ple when creating new dictionaries seems contrary to the advice provided when standard

dictionaries, such as those found in Hart’s (2000) DICTION (which were deductively devel-

oped independent of any particular type of narrative text) are used. This advice may also be

problematic for management theorists who are interested in a deductive, rather than induc-

tive, analysis of narrative texts. As illustration, Wade, Porac, and Pollock (1997) examined

the justification of CEO pay, explaining their content analysis procedure by noting, ‘‘We

322 Organizational Research Methods

first created a starting membership array for a concept by listing words and phrases that a

priori seemed to fit the category’s definition’’ (p. 649). Given somewhat unclear or poten-

tially conflicting recommendations in the literature, our second goal is to provide sugges-

tions concerning the creation and use of word lists in a manner that is compatible with the

use of CATA for organizational research. Consequently, we highlight how content, discri-

minant, external, and predictive validity can be used in empirical efforts using CATA to

encourage more rigorous efforts when content analysis is applied in organizational research.

Our third contribution is to provide an illustration of our CATA suggestions using the

construct of entrepreneurial orientation. Although entrepreneurship scholars have called for

the incorporation of content analysis to measure entrepreneurial orientation (Lyon, Lump-

kin, & Dess, 2000), relatively few studies have used content analysis to assess elements of

this construct (e.g., Chen & Hambrick, 1995). Given that content analysis may be an espe-

cially useful tool for examining adolescent theories (theories that have not been developed

or tested adequately, Sonpar & Golden-Biddle, 2008), we believe that illustrating how

entrepreneurial orientation can be assessed through content analysis of shareholder letters

is an especially attractive approach that offers value to entrepreneurship scholars as well

as to the broader management field.

Following our three primary contributions, we first review methods for assessing con-

struct validity when using content analysis and assess the extent to which such techniques

have been used in research to date. Second, we suggest a number of possible procedures to

enhance validity when using CATA. Third, we provide an illustration using the construct of

entrepreneurial orientation, concluding with implications for future research efforts.

Examining Validity in Content Analysis

To examine validity in studies using content analysis, we systematically investigated the

extent to which different types of validity applicable to research using content analysis were

assessed. We based our analyses on the sample of 98 studies in the recent review of Duriau

et al. (2007) in Organizational Research Methods of content analysis usage in management

research. In evaluating these studies, we assessed four types of validity commonly

addressed in the management literature: content, external, discriminant, and predictive

validity. A brief description of each type of validity, a description of how we coded each

type of validity, and the results of our investigation follow. A summary of our coding

results from these studies is displayed in Table 1.

Content validity is the degree to which a measure encapsulates the full domain of a par-

ticular construct (Nunnally & Bernstein, 1994). We coded studies as addressing content

validity if authors evaluated content analysis procedures for their appropriateness of repre-

senting the full domain of their study’s constructs. For example, we found that 79 of 98

(81%) studies relied on deductive procedures, based on a priori theory, when creating their

coding schemes. Another example of this evaluative process is having content experts judge

the adequacy of words in a content analysis dictionary to represent a particular construct.

We found that 21 studies of the 98 (21%) created word lists or dictionaries to represent their

study’s constructs, 11 of 98 studies (11%) described the process whereby authors selected

words to represent the study’s constructs, but only 5 of the 98 (5%) studies addressed


content validity through evaluating the adequacy of chosen words to represent other con-

structs. These five studies described varying processes to demonstrate content validity. For

instance, three of the five studies used expert judges to ensure that the chosen words fit the

construct of interest (e.g., Doucet & Jehn, 1997; Ferrier, 2001; Mossholder, Settoon, Harris,

& Armenakis, 1995), while the other two studies analyzed the meaning of selected diction-

ary words in actual sentences from the study’s text to ensure chosen words reflected the

meaning of the construct (e.g., Abrahamson & Park, 1994; Wade et al., 1997). In sum, while

it seems that most studies relied on deductive procedures, little detail was provided concern-

ing how authors selected and validated coding schemes.

External validity is the ability to generalize findings across multiple settings (Cook &

Campbell, 1979). We coded studies that applied a content analysis coding scheme to more

than one sample to evaluate the study’s predicted relationships as demonstrating external

validity. Our examination of the 98 studies revealed that only 12 (12%) demonstrated exter-

nal validity by including multiple samples in their research design. Given that most content

analysis studies rely on archival data, it is perhaps surprising that so few studies examined

external validity.

Discriminant validity involves the extent to which a construct is distinct from other con-

structs (Campbell & Fiske, 1959). We coded studies as addressing discriminant validity if

they examined dimensionality, created word lists for multiple constructs with the hopes of

establishing dimensionality, or used exploratory or confirmatory factor analysis to potentially

distinguish constructs. We found that only 5 studies of the 98 (5%) assessed the dimension-

ality or provided any procedures to assess discriminant validity of their study’s constructs.

Predicative, or nomological, validity is the extent to which measures predict other con-

structs to which they are theoretically linked (Cronbach & Meehl, 1955). We coded studies

as addressing predictive validity if the relationship between content analysis constructs and

other theoretically related constructs were analyzed. We discovered that 41 of the studies in

the review of Duriau et al. (2007; 42%) displayed predictive validity.

Table 1

Examination of Validity in Content Analysis Research

Content Analysis Studies Used by

Duriau et al. (2007)

Entrepreneurial Orientation Studies

Using Content Analysis

Number of studies 98 9

Deductive approach 79 of 98 (81%) 9 of 9 (100%)

Evaluated content validity of

word dictionaries

5 of 98 (5%) 0 of 9 (0%)

Computer-aided text analysis

(CATA)

24 of 98 (25%) 0 of 9 (0%)

Assessment of multiple

samples

12 of 98 (12%) 0 of 9 (0%)

Assessment of construct

dimensionality

5 of 98 (5%) 0 of 9 (0%)

Assessment of predicative

Validity

41 of 98 (42%) 9 of 9 (100%)

Citation support 36 of 98 (37%) 9 of 9 (100%)


To complement our examination of content analytic studies highlighted by Duriau et al.

(2007), we identified nine additional studies from the entrepreneurship and strategic man-

agement literatures that used content analysis to examine dimensions of entrepreneurial

orientation (the construct we examine in our illustration of potential procedures to assess

construct validity when using content analysis). Similar to our findings from the Duriau

et al. (2007) studies, we found that various forms of validity were not consistently addressed

in the nine entrepreneurial orientation studies. Although 8 of 9 (89%) studies used key word

dictionaries, only two described how words were selected for the dictionary and neither of

these demonstrated content validity by providing an explanation of the process for word

selection to accurately represent the studies’ constructs. Additionally, no study demon-

strated external validity through the use of multiple samples. Although discriminant validity

was not always applicable (some studies examined only one dimension of entrepreneurial

orientation), it is perhaps disappointing that no study addressed dimensionality given that

entrepreneurial orientation has been theorized as a multidimensional construct (Covin &

Slevin, 1989; Lumpkin & Dess, 1996). We were encouraged to find that all nine studies

demonstrated predictive validity by analyzing the relationship between entrepreneurial

orientation and firm performance, given that such linkages have consistently been theorized

in the entrepreneurship literature (Lumpkin & Dess, 1996).

In addition to assessing the extent to which validity was addressed in content analytic

studies, we also wanted to gauge the extent to which organizational researchers relied on

established content analysis methodology texts or other relevant sources when applying

content analysis to a particular question of interest. Thus, we examined the reference

sections of the 98 content analysis articles reviewed in Duriau et al. (2007), as well as the

9 content analysis articles measuring elements of entrepreneurial orientation, to assess

whether scholars referred to other content analysis research studies (scholarly articles

on the topic of content analysis) or statistical texts (books) about content analysis. Our

analysis revealed that content analysis studies often describe their procedures but fre-

quently do not cite other content analysis research in developing these procedures. For

instance, 63% of the content analysis studies identified by Duriau et al. (2007) did not cite

any content analysis–related work such as content analysis textbooks or other content

analysis studies that used a particular content analysis approach. Of those studies from

the Duriau et al. (2007) sample that did include a content analysis citation, the most fre-

quent was to Holsti (1969) at 10% followed by Weber (1990) at 9% and Jauch, Osborn,

and Martin (1980) at 7%. There are two potential interpretations of our findings. The gen-

eral lack of citing established content analysis work might suggest that management

researchers are not incorporating the most rigorous standards when applying content anal-

ysis to management research questions. Alternatively, researchers may in fact be incor-

porating established work into their techniques but provide little detail to explain their

process or guide future efforts.

Overall, our analyses show that no single study used all of the potential methods to assess

validity that we reviewed. The general lack of content validity process descriptions, as well

as infrequently demonstrated external, discriminant, and predictive validity and citation

support in developing content analysis procedures, suggest the need to propose validity

guidelines for studies using content analysis. Accordingly, in the next section, we suggest

a number of possible procedures to address issues of validity when using CATA.


Construct Validation Using CATA: Recommended Procedures

In this section, we suggest a number of procedures for validating constructs using content

analysis. Our goals are twofold. First, we seek to provide content analysis procedures that

are consistent with the applied nature of organizational research. Second, our aim is to pro-

vide a number of possibilities for researchers seeking to enhance construct validity when

incorporating content analysis. We believe our procedures can be applied to any construct

(unidimensional or multidimensional), as long as content analysis can be used to convey

meaning of naturally occurring phenomena (Van Maanen, 1979). However, we realize that

differences in research goals, as well as practical considerations (e.g., journal page length

restrictions), will make it unlikely that all of these procedures will be incorporated into any

given study. We begin by discussing elements related to content validity, followed by exter-

nal validity, reliability, dimensionality, and predictive (nomological) validity. We do not

mean to suggest that researchers who do not follow each of these techniques risk conducting

a study that lacks validity. Rather, we build on the guidelines of others who suggest that

more than one approach is desirable when seeking to establish validity (e.g., Campbell &

Fiske, 1959; Scandura & Williams, 2000). Furthermore, we offer a number of possibilities

to provide clear guidance for organizational researchers interested in examining validity

when using content analysis in general, and specifically the CATA approach we outline and

illustrate below. A summary of our proposed procedures is provided in Table 2.

Content Validity

Construct measurement begins with a sound theoretical definition of the concept of inter-

est. Thus, content validity involves an assessment of measures by experts/judges or a pro-

cess capable of examining a match between theoretical definition and empirical

measurement, which is demonstrated when a measurement procedure properly samples the

theoretical content domain of a construct (Nunnally & Bernstein, 1994). We advocate an

approach to content analysis using CATA with single words as the unit of analysis. This

type of content analysis is well established in psychological research traditions and is based

on the assumption that the words people use provide valuable insight related to the thought

processes reflected in narrative texts (Pennebaker, Mehl, & Niederhoffer, 2003). In the fol-

lowing sections, we provide guidance for the use of both deductive and inductive

approaches to CATA.

Deductively derived word lists. We recommend CATA begin with a deductive approach

that has been used by a number of others in the management field, where word lists are first

developed (independent of organizational documents) and then applied to meaningful firm-

level narratives (e.g., Porac, Wade, & Pollock, 1999; Wade et al., 1997).1 A deductive pro-

cess requires theory to design the coding scheme (Potter & Levine-Donnerstein, 1999).

Despite the prevalence of this technique in the literature (e.g., Dowling & Kabanoff,

1996; Weber, 1990), little guidance exists concerning how this method could be applied

to a variety of constructs germane to management research when existing dictionaries do

not adequately capture the construct’s content domain. We agree with others who advocated

deriving words from prior theory when possible (Dowling & Kabanoff, 1996; Weber, 1990)


and agree with organizational scholars who argued that content analysis might be an espe-

cially useful tool for examining adolescent theories (Sonpar & Golden-Biddle, 2008). How-

ever, we believe the only necessary condition is that researchers can create plausible word

lists to measure the construct of interest in relevant organizational narratives.

We suggest a four-step process for this deductive approach to content analysis. First,

researchers develop a working definition of the construct, preferably derived from existing

literature. Second, an initial assessment of the dimensionality of the construct of interest

should be made. Third, a list of key words that correspond to the formal definition is devel-

oped to capture the construct of interest. If the construct is conceptualized as multidimen-

sional, multiple discrete word lists are created for each subdimension. When developing

word lists, examining previously validated scales can yield insight and word suggestions.

An exhaustive list of synonyms should also be considered for the referent construct (or mul-

tiple lists if the construct is multidimensional). For example, Rodale’s (1978) The Synonym

Finder has been used in previous semiotic analysis (e.g., Markel, 1998). Fourth, word lists

are validated and rater reliability assessed (cf. Deephouse, 1996, 2000). Ideally, word lists

are assessed by a judge or panel of experts who are knowledgeable in the specific topic. We

Table 2

Recommended Procedures to Enhance Construct Validity When Using CATA

Deductive content validity

1. Create working definition of construct of interest (use a priori theory when possible)

2. Initial assessment of construct dimensionality based on existing literature

3. Develop an exhaustive list of key words from the formal definition to capture the construct of interest. (If

the construct is hypothesized to be multidimensional, multiple discrete word lists should be created for each

subdimension)

4. Validate word lists using content experts and assess rater reliability

Inductive content analysis

1. Identify commonly used words from narrative text of interest using DICTION or other CATA software

2. Identify or create a working definition of the construct of interest to guide word selection

3. Identify words that match the construct of interest

4. Establish initial interrater reliability

5. Refine and finalize word lists

Assess external validity

1. Select appropriate samples and relevant narrative texts to examine construct of interest

2. Compare two relevant samples when possible

Ensure reliability

Assure reliability by analyzing texts using a computer-aided technique such as DICTION, TEXTPACK,

WordStat, NVivo, or another that computes word counts (see Neuendorf, 2002, for a review of computer-

aided text analysis and associated programs)

Assess dimensionality (for multidimensional constructs)

Assess construct dimensionality using visual inspection of the correlation matrix. If dimensions are uncor-

related, they might be assessing different constructs and dimensions might exhibit problems of convergent

validity. If dimensions are correlated over .5, the construct may not be multidimensional. If dimensions

exhibit too high a correlation, consider collapsing subdimensions to form a single measure (or less

subdimensions)

Assess predictive (nomological) validity

Examine ability to predict theoretically related variables not captured via content analysis using regression,

structural equation modeling, or other relevant statistical technique


recommend Holsti’s (1969) method for assessing interrater reliability using the following

formula (PAO¼ 2A/nAþ nB) where PAO is proportion agreement observed, A is the number

of agreements between the two raters, and nA and nB are the number of words coded by the

two raters. Although there is no generally accepted ‘‘rule of thumb’’ for interrater reliability

coefficients analogous to the .70 heuristic for coefficient a, Riffe, Lacy, and Fico (2005)

and Krippendorff (2004) suggested interpreting values greater than .80, and Ellis (1994)

indicated coefficients between .75 and .80 as indicative of high reliability.

Inductively derived word lists. Although we advocate a deductive approach as a starting

point for our CATA approach, we believe there is value in exploring an inductive approach

to content analysis where narrative texts of interest are examined for potential words not

identified through deductive procedures alone. This approach is consistent with previous

organizational scholarship that has used a combination of inductive and deductive methods

in developing word lists (i.e., Doucet & Jehn, 1997; Kabanoff, Waldersee, & Cohen, 1995).

Such an approach is also consistent with the call for more engaged scholarship that contends

that knowledge is best produced by a cooperative arrangement between scholars who are

theory driven and practitioners who are closer to the phenomena; we believe that incorpora-

tion of deductive and inductive content analysis helps to bridge the knowledge transfer

problem characteristic of the divide between theory and practice (Van De Ven & Johnson,

2006).

We suggest a five-step process for an inductive approach to CATA content analysis.

First, a comprehensive list of commonly used words in the narrative of interest should be

identified. Second, scholars should identify or create a working definition of the construct

of interest. In step 3, the authors should examine words from the list of frequently used

words in the document of interest and assess whether they match the construct to be mea-

sured. In step 4, authors (or an additional rater) should independently examine the lists to

identify words associated with the construct of interest and then calculate an initial interra-

ter reliability (using the same Holsti [1969] method for assessing interrater reliability as

with the deductive analysis). Establishing reliability at this stage is important to assess gen-

eral agreement as to words that represent the construct of interest. In step 5, the list may be

refined by a discussion of potential words identified by the coders through an iterative pro-

cess until both raters agree that a particular word can be expected to meaningfully relate to

the construct of interest in the narrative. For multidimensional constructs, the inductive pro-

cedure could potentially yield additional words that relate to specific dimensions of a con-

struct. Conversely, the inductive analysis might also yield new dimensions not conceived

by previous efforts.

External Validity

External validity continues to be a key concern in management research methods (Savall,

Zardet, Bonnet, & Peron, 2008; Scandura & Williams, 2000), and techniques for establishing

external validity can be readily applied to research using content analysis (Neuendorf, 2002).

External validity refers to the confidence for which an inference regarding a causal relation-

ship can be generalized across different types of persons, settings, and times (Cook &

Campbell, 1979). For researchers using content analysis, two key decisions should be


considered carefully. The first decision involves the selection of narrative texts for analysis.

Selection of appropriate documents for content analysis is critical to ensure that the con-

struct can reasonably be expected to be detected in a given narrative. Thus, a fit between

research question and the type of document used for CATA should be considered prior

to the coding of documents. Examples of fit between document type and research questions

include mission statements used to examine organizational philosophy (e.g., Palmer &

Short, 2008; Pearce & David, 1987) and political speeches used to investigate elements

of espoused leadership (e.g., Bligh et al., 2004a, 2004b). The most commonly used narrative

text in the management literature, however, is the CEO’s letter to shareholders in annual

reports (Duriau et al., 2007), and these narratives have been analyzed to examine research

topics such as corporate strategy (Bowman, 1984) and managerial attention (D’Aveni &

MacMillan, 1990). Overall, we believe the key consideration is that the narrative text being

analyzed should provide an adequate setting with the construct that content analysis is being

used to detect (Krippendorff, 2004).

A second decision involves selecting an adequate sampling frame. Sampling decisions

are critical for establishing generalizability of study findings, but the decisions made by

organizational researchers concerning selection of a particular sampling frame have often

lacked adequate justification (Short, Ketchen, & Palmer, 2002). In some cases, practical

concerns may encourage reliance on certain samples such as the common usage of large

firms in the strategic management literature to allow pairing narrative documents with

archival firm performance data obtained from databases (e.g., COMPUSTAT). Regardless

of the sample used, similar to selection of the narrative text, researchers should be careful to

select a sample that is guided by theory and provides an adequate match with the research

question of interest (Short et al., 2002).

A comparison of different sampling frames provides additional insight into research

questions of interest. For example, use of the same measurement and analysis on a different

population constitutes an empirical generalization and a key form of replication (Tsang &

Kwan, 1999). Our review of content analysis studies suggests that demonstration of this

type of external validity is fairly uncommon in organizational research (12% of studies

examined by Duriau et al., 2007). This is problematic given the enthusiastic encouragement

for replication and triangulation in the management field (Hambrick, 2007; Scandura &

Williams, 2000; Singh, Ang, & Leong, 2003), and such efforts seem ideal for research using

content analysis.

Reliability

Reliability refers to the consistency achieved in construct measurement in terms of its

stability, dependability, and predictability (Kerlinger & Lee, 2000). To demonstrate ade-

quate reliability, we recommend the use of CATA as it has the advantages of greater relia-

bility and is less tedious than when content analysis is performed by human coders

(Dowling & Kabanoff, 1996). Additionally, the use of CATA exhibits high test–retest relia-

bility when examining transcripts of individuals over time (Schnurr, Rosenberg, Oxman, &

Tucker, 1986). Comparisons between word-count and human-scored content analysis have

found that CATA techniques are more accurate (Rosenberg, Schnurr, & Oxman, 1990). A


number of CATA programs are now available commercially such as DICTION, TEXT-

PACK, WordStat, or NVivo (see Neuendorf [2002] for a review of associated programs).

Dimensionality

Dimensionality refers to the association of measurement to a single construct; in the case

of multidimensional constructs, each dimension should be associated with a single factor to

ensure adequate discriminant validity (Nunnally & Bernstein, 1994). We recommend cre-

ating multiple word lists when constructs of interest are conceptualized as multidimen-

sional. When using content analysis, researchers can examine content analysis results

from multiple word lists and assess construct dimensionality simply by using visual inspec-

tion of the correlation matrix comparing multiple word lists (cf. Hair, Anderson, Tatham, &

Black, 1998). If dimensions are uncorrelated, researchers might consider whether they are

actually assessing different constructs. If results demonstrate (a) excessively high correla-

tions (more than .80), (b) no correlations, or (c) negative correlations (when none are

hypothesized), Hair et al. (1998) recommended conducting additional analyses to assess

dimensionality. For example, if dimensions exhibit too high of a correlation, consider col-

lapsing subdimensions to form a single measure, or conduct a factor analysis of word lists to

suggest fewer dimensions of a particular construct of interest.

Predictive Validity

Predictive validity (often dubbed nomological or criterion-related validity) refers to the

extent to which the operationalization predicts other constructs consistent with theoretically

derived a priori expectations (Cronbach & Meehl, 1955). It is demonstrated when con-

structs of interest are linked with others that are theoretically related. What sets this type

of validity apart from others is its emphasis on the role of theory and accompanying hypoth-

esis tests (Kerlinger & Lee, 2000). Unfortunately, predictive validity has often been under-

used in studies relying on content analysis (Neuendorf, 2002; Weber, 1990). To avoid issues

associated with common method variance, we encourage researchers to examine dependent

variables not captured via content analysis. In the strategic management literature, for

example, organizational performance operationalized with objective, archival data have

generally been the key dependent variable of interest (Meyer, 1991). Predictive validity can

be assessed using regression, structural equation modeling, or other accepted methodologi-

cal research methods associated within a particular research stream.

Validating CATA in Organizational Research: An Illustration Using

Entrepreneurial Orientation

To illustrate the potential application of our suggestions for construct validation when

using content analysis, we examine the construct of entrepreneurial orientation (Covin &

Slevin, 1988, 1989; Lumpkin & Dess, 1996). We believe this construct is attractive for a

number of reasons. First, although research in entrepreneurship in general has been charac-

terized by low paradigmatic development (Ireland, Reutzel, & Webb, 2005; Ireland, Webb,


& Coombs, 2005), entrepreneurial orientation has become a central construct in both the

strategic management and entrepreneurship literatures (Lyon et al., 2000; Tang, Tang, Mar-

ino, & Zhang, & Li, 2008). For example, according to scholar.google.com, Lumpkin and

Dess’s (1996) seminal article that appeared in Academy of Management Review has been

cited more than 1,000 times.

Despite the prevalence of the entrepreneurial orientation construct, numerous different

measures are used across this research stream. In many cases, scales are modified by drop-

ping (or adding) items without providing theoretical justification or demonstrations of psy-

chometric adequacy (e.g., Atuahene-Gima & Ko, 2001; George, Wiklund, & Zahra, 2005;

Slater & Narver, 2000). A meta-analysis of 51 entrepreneurial orientation studies by Rauch,

Wiklund, Frese, and Lumpkin (2004) reported that different variations (consisting of alter-

native or additional dimensions) of the foundational scales (e.g., Miller, 1983; Covin & Sle-

vin, 1989) were used in almost half of the studies, concluding that future research needs

more reliable and valid scales of the dimensions of entrepreneurial orientation (including

alternative approaches to measurement). Lyon et al. (2000) specifically advocated the use

of content analysis to measure entrepreneurial orientation.

Despite the espoused benefits of content analysis, its use in entrepreneurship studies has

been relatively sparse (Chandler & Lyon, 2001). However, content analysis has been

demonstrated as a useful approach to measuring aspects of entrepreneurial orientation

(e.g., Chen & Hambrick, 1995), and nine articles thus far have used content analysis to mea-

sure at least some elements of the construct. Yet, few studies have examined all five dimen-

sions outlined in Lumpkin and Dess’s (1996) Academy of Management Review article (see

Hughes & Morgan [2007] for a rare exception), and no study to date has used content anal-

ysis to link all five dimensions to firm performance in a single study. Thus, we believe

entrepreneurial orientation is an appropriate construct to demonstrate how content analysis

can make a methodological contribution to aid substantive research questions in the entre-

preneurship literature, especially when measurement has posed challenges in the past.

Furthermore, we believe that illustrating how entrepreneurial orientation can be assessed

with content analysis of shareholder letters is an especially attractive approach and offers

value to management researchers.

Content Validity

We began by identifying a formal definition of the entrepreneurial orientation construct

from the literature. We relied on the Lumpkin and Dess’ (1996) definition of entrepreneur-

ial orientation as the ‘‘processes, practices, and decision-making activities that lead to new

entry’’ (p. 136). Our second step involved an initial assessment of the content domain and

dimensionality of the entrepreneurial orientation construct based on an extensive review of

the literature. In the case of entrepreneurial orientation, considerable conceptual and

empirical work supports a five-dimension conceptualization of entrepreneurial orientation:

autonomy, competitive aggressiveness, innovativeness, proactiveness, and risk taking

(Lumpkin & Dess, 1996). In our third step, we developed an exhaustive list of words to cap-

ture each of the five entrepreneurial orientation dimensions. Had theory suggested entrepre-

neurial orientation as a unidimensional construct, we would have created a single word list;

because entrepreneurial orientation has been a priori theorized as multidimensional, we


created a discrete and exhaustive word list for each of its theoretically based dimensions

from The Synonym Finder (Rodale, 1978). As an example, the dictionary for autonomy

included variants of the word (autonomous), as well as synonyms (self-directing). It is

important that word lists used are exhaustive, mutually exclusive, and at the appropriate

level of measurement, so that each word can be associated with one and only one dimension

in the case of multidimensional constructs (Neuendorf, 2002).

Word lists were validated in step 4 using a multistep process with raters. In our example,

two raters examined the words generated by The Synonym Finder and compared them with

the theoretical definition of entrepreneurial orientation, deleting words that did not match the

specified dimension and adding words that could be associated with the construct. In our case,

both raters had published in the entrepreneurship literature and one scholar was blind to the

study’s purpose and not part of the author team. Of the 717 words generated by The Synonym

Finder and the 43 words added by our two raters, 244 words were selected by both raters as

representative of entrepreneurial orientation and were retained for subsequent analyses

(autonomy¼ 36 words, innovativeness¼ 86, proactiveness¼ 27, competitive aggressiveness

¼ 58, risk taking ¼ 37, respectively; the final word list is presented in Table 3).

To demonstrate interrater reliability of our nominal coding scheme (agree/disagree

between the raters), we used Holsti’s (1969) method discussed previously. Our observed

reliabilities ranged from .75 to .88 (reliabilities for each of the dimensions are presented

in Table 6), demonstrating consistency between our raters.

We supplemented our deductive analysis with an inductive procedure based on the word

choices evident in CEO letters to shareholders (the document of interest in our illustrative

example). First, we identified a comprehensive list of commonly used words from our sam-

ple of shareholder letters. Similar to others who have applied an inductive approach to dic-

tionary development (e.g., Abrahamson & Park, 1994; Doucet & Jehn, 1997), we generated

an exhaustive list of frequently used words within all texts across both of our samples. DIC-

TION was helpful in this regard as it calculates an ‘‘insistence score,’’ which assesses the

degree to which a text is dependent on often-repeated words. To calculate this score, DIC-

TION identifies all words mentioned three or more times within a particular text. We

extracted these words and created an exhaustive list of frequently used words across all

texts from our two samples. This analysis resulted in 3,331 frequently used words found

in our shareholder letter samples. Second, we looked for a general definition/explanation

of entrepreneurial orientation that could guide our inductive coding. Such a definition could

be generated by the researchers interested in conducting an inductive analysis or culled

from previous work. Given the prevalence of research in entrepreneurial orientation, we

used The Blackwell Encyclopedia of Management: Entrepreneurship (Hitt & Ireland,

2005) to guide us toward a general understanding of the nature of the entrepreneurial orien-

tation construct. The section of the encyclopedia included a general discussion of the entre-

preneurial orientation construct as well as specific definitions. For example, the authors of

this section (Lumpkin & Dess, 2005) noted (p. 104):

An early definition of entrepreneurial orientation was provided by Miller, who stated an entre-

preneurial firm is one that ‘‘engages in product market innovation, undertakes somewhat risky

ventures and is first to come up with ‘proactive’ innovations, beating competitors to the

punch.’’ (1983, p. 770)


Table 3

Word Lists for Entrepreneurial Orientation Dimensions

Entrepreneurial Orientation

Dimension Content Analysis Words With Expert Validation

Autonomy At-liberty, authority, authorization, autonomic, autonomous, autonomy, decon-

trol, deregulation, distinct, do-it-yourself, emancipation, free, freedom, free-

thinking, independence, independent, liberty, license, on-one’s-own, preroga-

tive, self-directed, self-directing, self-direction, self-rule, self-ruling, separate,

sovereign, sovereignty, unaffiliated, unattached, unconfined, unconnected,

unfettered, unforced, ungoverned, unregulated

Innovativeness Ad-lib, adroit, adroitness, bright-idea, change, clever, cleverness, conceive,

concoct, concoction, concoctive, conjure-up, create, creation, creative, crea-

tivity, creator, discover, discoverer, discovery, dream, dream-up, envisage,

envision, expert, form, formulation, frame, framer, freethinker, genesis, genius,

gifted, hit-upon, imagination, imaginative, imagine, improvise, ingenious,

ingenuity, initiative, initiator, innovate, innovation, inspiration, inspired,

invent, invented, invention, inventive, inventiveness, inventor, make-up,

mastermind, master-stroke, metamorphose, metamorphosis, neoteric, neoter-

ism, neoterize, new, new-wrinkle, innovation, novel, novelty, original, ori-

ginality, originate, origination, originative, originator, patent, radical, recast,

recasting, resourceful, resourcefulness, restyle, restyling, revolutionize, see-

things, think-up, trademark, vision, visionary, visualize

Proactiveness Anticipate, envision, expect, exploration, exploratory, explore, forecast, fore-

glimpse, foreknow, foresee, foretell, forward-looking, inquire, inquiry, inves-

tigate, investigation, look-into, opportunity-seeking, proactive, probe,

prospect, research, scrutinization, scrutiny, search, study, survey

Competitive aggressiveness Achievement, aggressive, ambitious, antagonist, antagonistic, aspirant, battle,

battler, capitalize, challenge, challenger, combat, combative, compete, comp-

eter, competing, competition, competitive, competitor, competitory, conflict-

ing, contend, contender, contentious, contest, contestant, cutthroat, defend,

dog-eat-dog, enemy, engage, entrant, exploit, fierce, fight, fighter, foe, intense,

intensified, intensive, jockey-for-position, joust, jouster, lock-horns, opponent,

oppose, opposing, opposition, play-against, ready-to-fight, rival, spar, strive,

striving, struggle, tussle, vying, wrestle

Risk taking Adventuresome, adventurous, audacious, bet, bold, bold-spirited, brash, brave,

chance, chancy, courageous, danger, dangerous, dare, daredevil, daring,

dauntless, dicey, enterprising, fearless, gamble, gutsy, headlong, incautious,

intrepid, plunge, precarious, rash, reckless, risk, risky, stake, temerity, uncer-

tain, venture, venturesome, wager

Additional inductively

derived words

Advanced, advantage, commercialization, customer-centric, customized,

develop, developed, developing, development, developments, emerging,

enterprise, enterprises, entrepreneurial, exposure, exposures, feature, features,

founding, high-value, initiated, initiatives, innovations, innovative, introduc-

tions, launch, launched, leading, opportunities, opportunity, originated, out-

doing, outthinking, patents, proprietary, prospects, prototyping, pursuing, risks,

unique, ventures

Source: Deductive word lists were developed with the aid of Rodale’s (1978) The Synonym Finder. Of the 717

words generated by The Synonym Finder and the 43 words added by our two raters, 244 words were selected by

both raters as representative of entrepreneurial orientation and were retained for subsequent analyses.


In step 3, two authors independently examined the lists to identify words associated with

entrepreneurial orientation and then calculated an interrater reliability (using the same Hol-

sti [1969] method for assessing interrater reliability as with the deductive analysis). Estab-

lishing reliability at this stage is important to assess general agreement as to words that

represent the construct of interest. Our initial interrater reliability for our inductive analysis

was .97. In step 4, the list was refined by a discussion of potential words identified by one

but not both authors through an iterative process until both authors agreed that a particular

word could be expected to meaningfully relate to the construct of entrepreneurial orienta-

tion in shareholder letters.

The identification of additional words using an inductive approach suggests the value of

both deductive and inductive methods when using CATA. Indeed, our inductive analysis

led to the identification of 41 additional words not generated by the deductive procedures.

To highlight the additional words from the inductive analysis, we present them separately

from the deductively defined lists in Table 3. The incorporation of an inductive approach

could also be used to identify the presence of additional entrepreneurial orientation dimen-

sions or the need for greater refinement of existing dimensions. For instance, one induc-

tively derived word, ‘‘commercialization’’ seems to capture unique elements of

entrepreneurship not fully developed in current conceptualizations of entrepreneurial orien-

tation. Commercialization is a process of converting knowledge into marketable products or

services and is an important aspect of entrepreneurship that is vital to new venture growth

(e.g., Kazanjian, 1988; Shane, 2004). Although this process is likely related to other entre-

preneurial orientation dimensions (e.g., proactiveness, risk taking, etc.), a firm’s propensity

to commercialize independent of being proactive or taking risks may represent a unique

dimension of its entrepreneurial orientation. Although we did not separate terms associated

with commercialization from other words identified through our inductive analysis, sub-

stantive efforts may benefit from an iterative process, where inductively derived words are

either incorporated into deductive lists or used to reveal new dimensions.

External Validity

We relied on shareholder letters as the organizational narrative that was particularly

applicable to the measurement of entrepreneurial orientation for a number of reasons. First,

letters to shareholders are excellent sources of managerial cognitions as they provide a

means for reconstructing perceptions and beliefs of authors (D’Aveni & MacMillan,

1990). Shareholder letters in particular signal the major topics and themes to which man-

agers attend (Barr, Stimpert, & Huff, 1992). Thus, annual report texts are useful for captur-

ing elements of top management’s values, beliefs, and ideologies (which include

entrepreneurial orientations). Second, the letter is the most widely read section of the annual

report (Courtis, 1982) and provides a forum for the CEO to voice thoughts on important

issues affecting the organization (Goodman, 1980). Indeed, empirical evidence suggests

that the CEO actively participates in the writing process of the letter (cf. Amernic, Craig,

& Tourish, 2007). Furthermore, the rhetoric in these texts has been related to organizational

actions and outcomes (Bowman, 1984; Michalisin, 2001).

For example, assertions of innovativeness (one dimension of entrepreneurial orientation)

in the letter to shareholders were related to firm reputation for innovation and the number of


trademarks applied for by the firm (Michalisin, 2001). Such results lend support to the

notion that communication in the shareholder letters reflects actual firm behaviors and

makes them particularly useful for examinations of entrepreneurial orientation.

We chose the S&P 500 for our sampling frame. There is recognition in the strategic

management literature that large established firms may benefit or even be forced to adopt

an ‘‘entrepreneurial mindset’’ in highly competitive landscapes (Bettis & Hitt, 1995) and

thus should undertake entrepreneurial behaviors to grow and survive (Zahra, 1991). With

growing interest in entrepreneurship within large firms, it is not surprising that entrepre-

neurial orientation has not been limited to the study of small or nascent firms, but rather,

sampling frames in previous studies have ranged from small private firms to very large

public firms drawn from the Fortune 500 (e.g., Hult, Snow, & Kandemir, 2003; Zahra,

1996) as well as public firms with more than US$25 million in sales (e.g., Sarkar, Echambadi,

& Harrison, 2001). Because the S&P 500 is comprised of publicly traded firms, the use of

this sample allows for the collection of additional variables (i.e., measures of organiza-

tional size and firm performance) through a secondary source without the threat of intro-

ducing common method variance. These measures, obtained from the COMPUSTAT

database, were used to demonstrate nomological validity of our measure of entrepreneur-

ial orientation. Our criterion for the use of a particular letter is that the firm was consis-

tently listed in the S&P 500 for 2001–2005, resulting in letters and data from 450 firms in

2005 (our year of data collection).

To allow for an assessment of external validity, we collected an additional sample com-

prised of small, high-growth firms to allow for comparisons with our sample of S&P 500

firms. We selected firms from the Russell 20001 stock index for this comparison group.

The 2,000 firms in this index are drawn from a larger Russell 30001 stock index that rep-

resents approximately 98% of the total value of the U.S. equity market (Russell Invest-

ments, 2008a). In contrast, the Russell 20001 firms represent only 10% of the total

value of the Russell 30001 stock index and as such provide a large sample of small, pub-

licly traded firms (Russell Investments, 2008b). Moreover, the Russell 20001 is considered

an important benchmark index for small valued companies in investment portfolios (Cheng,

Liu, & Qian, 2006; Reilly & Wright, 2002). Specifically, we selected the 500 highest

growth firms in this index. Following the method used by Inc. magazine to select its top

500 private entrepreneurial firms, we calculated the average annual revenue growth for

each firm in the Russell 20001 index for more than a five-year period (2001–2005) and

selected the top 500 high growth firms for our comparison sample. Average annual revenue

growth for our Russell 20001 sample was much higher than the S&P 500 sample (i.e.,

148% vs. 18%), suggesting that our Russell 20001 sample involved firms who collectively

pursue entrepreneurial activities that spur such revenue growth (and thus, a more entrepre-

neurial sample to compare with our S&P 500 sample). Similar to our S&P 500 sample, we

collected shareholder letters for 2005 and content analyzed the resulting 205 shareholder

letters with DICTION using our entrepreneurial orientation custom dictionaries.

For both samples, we conducted one sample t tests (compared to a test statistic of zero)

for each entrepreneurial orientation dimension to evaluate the presence of language consis-

tent with an entrepreneurial orientation in shareholder letters (a zero result indicates that

language consistent with an entrepreneurial orientation is not present in our sample of

shareholder letters). As shown in Table 4, all dimensions were significant suggesting that


language consistent with an entrepreneurial orientation was communicated in the share-

holder letters across both samples.

We also directly examined external validity with the two samples. Because of the differ-

ences in available shareholder letters between the two samples, we took a random set of 205

firms from our S&P 500 sample to ensure that differences between samples were not influ-

enced by sample size differences. To evaluate mean differences between the samples, we

conducted a one-way analysis of variance (ANOVA). Results of this analysis revealed

no significant differences in mean values in entrepreneurial orientation dimensions with the

exception of proactiveness, where higher mean levels were detected in the shareholder let-

ters of Russell 20001 firms (See Table 5).

Reliability

To enhance reliability, we relied on CATA as it minimizes the error from human coders

(such as rater fatigue or inadequate coder training). Similar to human coding schemes,

CATA generally assesses content via word usage (Morris, 1994). Relying on text to study

cognition assumes that the insights about the authors’ mental models can be detected

through the presence of, absence of, and frequency with which certain concepts are used

in text (Carley, 1997). CATA is advantageous in that multiple texts can be analyzed in min-

utes with near perfect reliability and without coder bias (Stevenson, 2001).

A number of software programs are available for content analysis. In our analyses, we

used DICTION (Hart, 2000) because it has been used in previous research examining man-

agement constructs such as charismatic leadership (Bligh et al., 2004a, 2004b). Further-

more, researchers have advocated DICTION as content analysis software with the

potential to measure a number of theoretically based constructs of interest to strategic man-

agement and entrepreneurship research (Short & Palmer, 2008). Using CATA, we analyzed

1,512 single-spaced pages of text in the S&P 500 and Russell 20001 samples.

Table 4

Evidence of Language Representing Entrepreneurial Orientation Dimensions in

Shareholder Letters of Firms in S&P 500 and Russell 20001

S&P 500 Sample Russell 20001 Sample

N Mean SD t Test N Mean SD t Test

Autonomy 453 1.01 2.11 10.23* 205 .65 1.07 8.70*

Competitive aggressiveness 453 3.13 2.99 22.25* 205 2.04 2.17 13.46*

Innovativeness 453 12.76 8.94 30.38* 205 9.13 7.75 16.86*

Proactiveness 453 2.95 3.20 19.60* 205 3.07 3.62 12.14*

Risk taking 453 1.11 2.05 11.50* 205 .78 2.02 5.49*

Inductively derived entrepreneurial orientation words 453 18.94 13.46 29.85* 205 15.22 10.85 20.03*

Note: The results of this table were based on computer-aided text analysis using the word lists for entrepre-

neurial orientation presented in Table 3.*p < .01.


Dimensionality

To assess the multidimensionality of entrepreneurial orientation, we conducted two addi-

tional steps. An underlying assumption in construct measurement is that measures should be

associated with a single concept for unidimensional measures; for constructs that are pro-

posed to be multidimensional, each dimension should be distinct from but related to the others

(Edwards, 2000). When using content analysis to assess construct validity, we propose exam-

ination of the correlation matrix of the DICTION scores to reveal significant but not perfect

correlations between dimensions for a multidimensional construct; high correlations would

provide evidence of a unidimensional construct. Table 6 displays the correlations for each

dimension of entrepreneurial orientation in our two samples, based on the bivariate correla-

tions of the word lists used to measure entrepreneurial orientation dimensions. These results

provide evidence that entrepreneurial orientation is a multidimensional construct. For the

S&P 500 firms, all dimensions were correlated at the p < .05 or greater level, with the excep-

tion of risk taking and innovativeness. Similar to the S&P 500 sample, the correlations among

dimensions in the Russell 20001 sample were all positive except for the correlation between

autonomy and risk taking. Moreover, although not all dimensions in the Russell 20001 sam-

ple displayed a significant relationship with the other dimensions, each dimension was signif-

icantly related (at least p < .05) to at least one other dimension except risk taking. In both

samples, no correlations were higher than .40. Thus, as all measures were correlated less than

.5 with any other measure, evidence from our sample is consistent with Lumpkin and Dess’s

(1996) multidimensional conceptualization of entrepreneurial orientation.

Predictive Validity

In entrepreneurial orientation research, the most examined causal relationship of interest

is between entrepreneurial orientation and firm performance, where most studies assume a

Table 5

ANOVA Comparisons of S&P 500 to Russell 20001 Firms on Entrepreneurial

Orientation Dimensions

Entrepreneurial Orientation Dimension

S&P 500 Sample

(N ¼ 205)

Russell 20001 Sample

(N ¼ 205) F Test

Autonomy .61 .57 .16

Competitive aggressiveness 1.87 1.64 2.43

Innovativeness 7.31 7.15 .15

Proactiveness 1.67 2.61 14.68**

Risk taking .76 .53 4.00*

Inductively derived entrepreneurial

orientation words

11.42 14.06 9.16**

Total entrepreneurial orientation 20.54 21.90 3.05

Note: ANOVA ¼ analysis of variance. The results of this table were based on computer-aided text analysis

using the word lists for entrepreneurial orientation presented in Table 3. Entrepreneurial orientation dimensions

are standardized by the number of words in the shareholder letter.*p < .05.**p < .01.


direct and positive relationship (Rauch et al., 2004). To measure firm performance we relied

on Tobin’s Q, a measure that assesses the degree to which the stock market values a firm

relative to its replacement cost (Mehran, 1995) and is commonly used in the entrepreneur-

ship literature (e.g., Andrews & Welbourne, 2000; Dyer & Whetten, 2006). To allow for a

potential lag effect, we used a 2-year performance measure based on the average Tobin’s Q

for the years 2005 and 2006. To assess the relationship of the entrepreneurial orientation

dimensions with firm performance, we first controlled for organizational size using the

logarithm of full-time employees (Konrad & Mangel, 2000).

During the process of examining our shareholder letters, we noticed that there was con-

siderable variance in the length of these documents. The mean number of words used in our

sample of CEO letters to shareholders was 1,566 words with the shortest letter using only

126 words and the longest using 6,513 words. To control for such a wide discrepancy in text

length across our sample, we standardized our detected entrepreneurial orientation con-

struct words by dividing them by the total number of words found in the text for our tests

of predictive validity. This control method is similar to others used in content analytic stud-

ies and is appropriate when texts vary considerably in length (i.e., Doucet & Jehn, 1997;

Emrich, Brower, Feldman, & Garland, 2001).

Using stepwise regression, we found that the five entrepreneurial orientation dimensions

as a set significantly predicted performance for the S&P 500 and Russell 20001 samples.

As shown in Table 7, the magnitude of each dimension’s predictive power varied among

performance measures. For instance, in both samples, competitive aggressiveness and risk

Table 6

Intercorrelations of Entrepreneurial Orientation Dimensions to Assess

Dimensionality

Entrepreneurial Orientation Dimension 1 2 3 4 5 6

S&P 500 Sample

1. Autonomy .80

2. Competitive Aggressiveness .15** .75

3. Innovativeness .17** .38** .88

4. Proactiveness .10* .20** .30** .85

5. Risk Taking .11* .21** .06 .09* .83

6. Inductive Entrepreneurial Orientation (unique words) .18** .44** .57** .42** .13** .97

Russell 20001 Sample

1. Autonomy .80

2. Competitive aggressiveness .16** .75

3. Innovativeness .17** .39** .88

4. Proactiveness .01 .11** .19** .85

5. Risk Taking �.03 .13y .10 .07 .83

6. Inductive entrepreneurial orientation (unique words) .14* .32** .62** .36** .17* .97

Note: Holsti’s (1969) observed agreement proportions are shown in italics in the diagonal. The results of this

table were based on computer-aided text analysis using the word lists for entrepreneurial orientation presented in

Table 3.*p < .05.**p < .01.yp < .10.


taking displayed a significantly negative relationship with performance while innovative-

ness and proactiveness demonstrated a positive relationship to performance (although these

positive relationships were significant only in the S&P 500 sample). Autonomy was not sig-

nificantly related to performance in either sample. Table 8 shows the significant relation-

ships of the inductively defined entrepreneurial orientation words for both samples. In

Table 9, we examine both deductive and inductive lists and again find significance for both

samples. Overall, we found a fairly consistent pattern relating the entrepreneurial orienta-

tion measures to performance for both samples.

Our results with regard to mixed dimensionality and nonfindings vis-a-vis entrepreneur-

ial orientation dimensions and performance are not surprising and are consistent with find-

ings in this research stream (cf. Kreiser, Marino, & Weaver, 2002; Rauch et al., 2004). For

example, the relationship between proactiveness and performance has been equivocal. Sar-

kar et al. (2001) found a positive relationship between proactiveness and firm performance

(with proactiveness in smaller firms exhibiting the strongest relationship to performance).

Conversely, Morgan and Strong (2003) found a negative but nonsignificant relationship

between proactiveness and firm performance in their sample of medium-to-large high tech-

nology and industrial manufacturing firms. This pattern of results is also found in the

dimension of competitive aggressiveness. Ferrier (2001) reported a significant positive rela-

tionship with the number of competitive actions and firm performance, whereas Lumpkin

and Dess (2001) revealed that competitive aggressiveness was negatively but not signifi-

cantly related to firm performance. Additionally, in a rare study that included all five

dimensions of entrepreneurial orientation, mixed relationships were found with each

dimension and firm performance (Hughes & Morgan, 2007). In this analysis, proactiveness

and innovativeness were positively related to performance, while risk taking was negatively

Table 7

Hierarchical Regression Analyses for Deductively Defined Entrepreneurial

Orientation Measures Predicting Tobin’s Q: S&P 500 Firms and Russell 20001

Firms


1 2 1 2

Firm size (logarithm of employees) �.09 �.13** �.18** �.14*

Autonomy �.03 .02

Competitive aggressiveness �.08 �.11

Innovativeness .19** .15*

Proactiveness .12* .18*

Risk taking �.21** �.15*

R2 .01 .12 .03 .09

F for change in R2 3.46* 11.09** 6.05* 3.97**

Note: Table contains standardized regression coefficients. Measurement of entrepreneurial orientation dimen-

sions was based on computer-aided text analysis using the word lists for entrepreneurial orientation presented in

Table 3. Entrepreneurial orientation dimensions are standardized by the number of words in the shareholder

letter.*p < .05.**p < .01.


related and competitive aggressiveness and autonomy were positively but not significantly

related to firm performance. Our results incorporating deductive and inductive lists yielded

R2s of .14 and .13 for S&P 500 and Russell 20001 samples, respectively. These results rep-

resent some of the strongest relationships of entrepreneurial orientation to firm performance

to date (cf., Stam & Elfring, 2008; Tang et al., 2008; Walter, Auer, & Ritter, 2006).

Table 8

Hierarchical Regression Analyses for Inductively Based Entrepreneurial

Orientation Measure Predicting Tobin’s Q


1 2 1 2

Firm size (logarithm of employees) �.09 �.07 �.17* �.17*

Inductively derived entrepreneurial orientation words .21** .08

R2 .01 .05 .03 .04

F for change in R2 3.67 19.37** 5.93* 1.10

Note: Table contains standardized regression coefficients. Inductive entrepreneurial orientation is generated

from a computer-aided text analysis using a word list generated from an inductive analysis of CEO 2005

shareholder letters. This measure is standardized by the number of words in the shareholder letter. Tobin’s Q is

calculated using an average of performance for the years 2005 and 2006.*p < .05.**p < .01.

Table 9

Hierarchical Regression Analysis for Deductive and Inductive Entrepreneurial

Orientation Measures Predicting Tobin’s Q: S&P 500 Firms and Russell 20001

Firms


1 2 1 2

Firm size (logarithm of employees) �.09 �.10* �.18* �.14*

Autonomy �.03 .00

Competitive aggressiveness �.09 �.11

Innovativeness .15** .11

Proactiveness .10* .15*

Risk taking �.19** �.14*

Inductively derived entrepreneurial

orientation words (unique words)

.17** .20**

R2 .01 .14 .03 .13

F for change in R2 3.31 11.41** 6.05* 4.78**

Note: Table contains standardized regression coefficients. Measurement of entrepreneurial orientation dimen-

sions was based on computer-aided text analysis using the word lists for entrepreneurial orientation presented in

Table 3. Entrepreneurial orientation dimensions are standardized by the number of words in the shareholder

letter.*p < .05.**p < .01.


Discussion

We examine the use of methods to assess construct validity when using content analysis

and provide a number of useful procedures for future research efforts. By doing so,

researchers will be able to measure concepts of interest via content analysis for a variety

of subjects, where traditional survey or other measurement techniques are difficult to apply

while maintaining acceptable standards of validity. By implementing the checks on valid-

ity, we suggest, researchers can have more confidence in the inferences made from their

work. Failure to capitalize on such opportunities could result in inconsistent findings, lim-

iting our understanding of substantive topics of interest.

The contributions of our work should be viewed in light of its limitations. For example,

examinations of other types of validity would further strengthen and supplement the tests

we propose. Convergent validity (assessment of correlations between two measures of the

same construct) could potentially be assessed by comparing traditional survey measures of

entrepreneurial orientation with the content analytic measures we assessed. Given tradi-

tional challenges in response rates of CEOs, this could be a daunting task with sampling

frames such as the S&P 500. An alternative means to assess convergent validity would

be to compare CATA with results based on human coders. For example, research using

DICTION detected a high level of agreement between human coders and predefined DIC-

TION variables used to measure charismatic leadership (Bligh et al., 2004a, 2004b). Further

assessment of discriminant validity could be achieved when constructs such as entrepre-

neurial orientation are uncorrelated with dissimilar constructs such as consideration and

initiating structure dimensions of leadership (cf. Weber, 1990). In short, the limitations

of the current study represent fruitful avenues for future research.

The use of CATA may be accompanied by a number of trade-offs to human-coded anal-

ysis. For example, CATA is less context sensitive than human coders for detecting the mean-

ing of a word within a sentence. However, Rosenberg et al. (1990) found that computers

calculating word frequencies outperformed human coders in a project that assigned psychia-

tric patients to a diagnostic category based on a content analysis of their speech. Such a find-

ing suggests that the human coding advantage is less clear than some scholars might suspect.

Any advantage of human coding over CATA, however, must be weighed in light of the

advantages of CATA (i.e., perfectly reliable analysis and ability to analyze a large number

of texts quickly). Yet, our CATA approach may be less sensitive to examining the temporal

elements of word choice. For example, the unexpected finding that risk taking was signifi-

cantly and negatively related to firm performance in both of our samples could be a function

of word lists representing the risk-taking dimension of entrepreneurial orientation potentially

referring to past, present, and future risks all in the same shareholder letter. Such possibilities

suggest that complementing CATA with human coding could be helpful in discovering nuan-

ces concerning how a given construct is portrayed in a given sample of interest.

Our process is also useful for examining substantive research questions in a novel man-

ner that may allay previous concerns associated within some research streams. For example,

as mentioned in our illustration, tests linking entrepreneurial orientation to firm perfor-

mance have been equivocal (e.g., Smart & Conant, 1994; Zahra & Covin, 1995). Entrepre-

neurial orientation studies have often used perceptual measures of performance (e.g., Naldi,


Nordquist, Sjoberg, & Wiklund, 2007), but this practice could be problematic by introdu-

cing potential for common method variance. Our examination ameliorates such issues

because assessing entrepreneurial orientation through content analysis and performance via

COMPUSTAT provides an improvement over studies that have captured both constructs

through survey measures.

Extensions of our proposed procedures could yield further benefit. For example, in

exploring entrepreneurial orientation, we examined external validity by comparing two

samples using the same narrative text (i.e., CEO letters to shareholders). Our examination

of different samples using the same measures with two different populations constitutes an

empirical generalization, a key form of replication in the organizational sciences (Tsang &

Kwan, 1999). Analysis of different populations using an alternative data source would pro-

vide an additional form of empirical generalization. For example, entrepreneurship scholars

have called for content analysis of IPO prospectuses as a particularly useful data source

(Marino, Castaldi, & Dollinger, 1989). Previous content analyses found little link between

information found in CEO prospectus statements and organizational performance (e.g.,

Daily, Certo, & Dalton, 2005), but applying the dictionaries we created to prospectus state-

ments would provide an additional perspective. Other documents such as mission state-

ments have been advocated as important narratives that reflect organizational identity

(Palmer & Short, 2008), and examination of mission statements may also provide valuable

insight concerning articulation of entrepreneurial orientation when defining an organiza-

tion’s raison d’etre. Differences in firm performance have often been found based on indus-

try (e.g., Rumelt, 1991; McGahan & Porter, 1997) and strategic group membership (e.g.,

Short, Ketchen, Palmer, & Hult, 2007). Thus, comparisons between differences in specific

industry contexts (such as high-tech industries) and other industry contexts could provide an

additional extension for demonstrating external validity.

Conclusion

Measurement in the organizational literature is dynamic, and sophisticated analytic tech-

niques continue to emerge and evolve. Despite such advancements, little guidance exists in

terms of how to validate many abstract theoretical constructs that are assessed via content

analysis. It is our hope that our process will provide a useful blueprint toward that end and

begin a dialogue that will improve construct validation in the organizational sciences when

analyses use content analysis.

Note

1. We sought guidance from a number of statistical texts, as well as relevant management research, when

considering our proposal for content validity. We were also in contact with Roderick Hart, creator of one of the

most well-known computer-aided text analysis programs—DICTION. We were intrigued by Professor Hart’s

thoughts and inquired about his process when developing DICTION. Professor Hart noted the following:

Unlike many computerized language analysis programs that depend on users to create their

own dictionaries, DICTION is a theoretically derived program built around five key dimen-

sions of language behavior. Its search dictionaries were deductively derived, not inductively


derived, with the key factor being face validity—do the categories in fact measure what they

appear to measure? In content analysis, that is ultimately the most important kind of validity

because categories that look foolish upon inspection will not be used. Thus, before deploying

DICTION, users should inspect its dictionaries carefully. For example, if the words in its Praise

dictionary (astute, beautiful, and eminent) do not seem praiseworthy, the user should find

another program. All 10,000 of DICTION’s search words can be inspected in this way, impli-

citly demanding that users give a priori thought to the phenomena they are studying. DIC-

TION’s ability to tap exactly the same conceptually derived dimensions every time it is

used is therefore its greatest limitation and its greatest strength. (Hart, 2008, personal commu-

nication, June 11, 2008)

References

Abrahamson E., & Park C. (1994). Concealment of negative organizational outcomes: An agency theory per-

spective. Academy of Management Journal, 37, 1302-1334.

Amernic J., Craig R., & Tourish D. (2007). The transformational leader as pedagogue, physician, architect, com-

mander, and saint: Five root metaphors in Jack Welch’s letters to stockholders of General Electric. Human

Relations, 60, 1839-1872.

Andrews A. O., & Welbourne T. M. (2000). The people/performance balance in IPO firms: The effect of the

chief executive officer’s financial orientation. Entrepreneurship Theory and Practice, 25, 93-106.

Atuahene-Gima K., & Ko A. (2001). An empirical investigation of the effect of market orientation and entre-

preneurship orientation alignment on product innovation. Organization Science, 12, 54-74.

Bagozzi R. P., Yi Y., & Phillips L. W. (1991). Assessing construct validity in organizational research. Admin-

istrative Science Quarterly, 36, 421-458.

Barr P. S., Stimpert J. L., & Huff A. S. (1992). Cognitive change, strategic action, and organizational renewal.

Strategic Management Journal, 13, 15-36.

Bartholomew S., & Smith A. D. (2006). Improving survey response rates from chief executive officers in small

firms: The importance of social networks. Entrepreneurship Theory and Practice, 30, 83-96.

Baruch Y. (1999). Response rate in academic studies—A comparative analysis. Human Relations, 52, 421-438.

Bettis R. A., & Hitt M. A. (1995). The new competitive landscape. Strategic Management Journal, 16, 7-19.

Bligh M. C., Kohles J. C., & Meindl J. R. (2004a). Charisma under crisis: Presidential leadership, rhetoric, and

media responses before and after the September 11th terrorist attacks. The Leadership Quarterly, 15,

211-239.

Bligh M. C., Kohles J. C., & Meindl J. R. (2004b). Charting the language of leadership: A methodological inves-

tigation of President Bush and the crisis of 9/11. Journal of Applied Psychology, 89, 562-574.

Bowman E. H. (1984). Content analysis of annual reports for corporate strategy and risk. Interfaces, 14, 61-71.

Boyd B. K., Gove S., & Hitt M. A. (2005). Construct measurement in strategic management research: Illusion or

reality? Strategic Management Journal, 26, 239-257.

Campbell D. T., & Fiske D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod

matrix. Psychological Bulletin, 56, 81-105.

Carley K. M. (1997). Extracting team mental models through textual analysis. Journal of Organizational Beha-

vior, 18, 533-558.

Chandler G. N., & Lyon D. W. (2001). Issues of research design and construct measurement in entrepreneurship

research: The past decade. Entrepreneurship Theory and Practice, 25, 101-113.

Chen M. J., & Hambrick D. C. (1995). Speed, stealth and selective attack: How small firms differ from large

firms in competitive behavior. Academy of Management Journal, 38, 453-482.

Cheng Y., Liu M. H., & Qian J. (2006). Buy-side analysts, sell-side analysts, and investment decisions of money

managers. Journal of Financial and Quantitative Analysis, 4, 51-83.


Cook T. D., & Campbell D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings.

Boston: Houghton-Mifflin.

Courtis J. K. (1982). Private shareholder response to corporate annual report. Accounting and Finance, 22, 53-72.

Covin J. G., & Slevin D. P. (1988). The influence of organization structure on the utility of an entrepreneurial

top management style. Journal of Management Studies, 25, 217-234.

Covin J. G., & Slevin D. P. (1989). Strategic management of small firms in hostile and benign environments.

Strategic Management Journal, 10, 75-87.

Cronbach L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (pp. 443-507).

Washington, DC: American Council on Education.

Cronbach L. J., & Meehl P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

Daily C. M., Certo S. T., & Dalton D. R. (2005). Investment bankers and IPO pricing: Does prospectus infor-

mation matter? Journal of Business Venturing, 20, 93-111.

D’Aveni R., & MacMillan I. (1990). Crisis and the content of managerial communications: A study of the focus

of attention of top managers in surviving and failing firms. Administrative Science Quarterly, 35, 634-657.

Deephouse D. L. (1996). Does isomorphism legitimate? Academy of Management Journal, 39, 1024-1039.

Deephouse D. L. (2000). Media reputation as a strategic resource: An integration of mass communication and

resource-based theories. Journal of Management, 26, 1091-1112.

Doucet L., & Jehn K. A. (1997). Analyzing harsh words in a sensitive setting: American expatriates in commu-

nist China. Journal of Organizational Behavior, 18, 559-582.

Dowling G. R., & Kabanoff B. (1996). Computer-aided content analysis: What do 240 advertising slogans have

in common? Marketing Letters, 7, 63-75.

Duriau V. J., Reger R. K., & Pfarrer M. D. (2007). A content analysis of the content analysis literature in orga-

nizational studies: Research themes, data sources, and methodological refinements. Organizational

Research Methods, 10, 5-34.

Dyer W. G., & Whetten D. A. (2006). Family firms and social responsibility: Preliminary evidence from the

S&P 500. Entrepreneurship Theory and Practice, 30, 785-802.

Edwards J. R. (2000). Multidimensional constructs in organizational behavior research: An integrative analyti-

cal framework. Organizational Research Methods, 4, 144-192.

Ellis L. (1994). Research methods in the social sciences. Madison, WI: WCB Brown & Benchmark.

Emrich C. G., Brower H. H., Feldman J. M., & Garland H. (2001). Images in words: Presidential rhetoric, char-

isma, and greatness. Administrative Science Quarterly, 46, 527-557.

Fendt J., & Sachs W. (2008). Grounded theory method in management research. Organizational Research Meth-

ods, 11, 430-455.

Ferrier W. J. (2001). Navigating the competitive landscape: The drivers and consequences of competitive

aggressiveness. Academy of Management Journal, 44, 858-877.

George G., Wiklund J., & Zahra S. A. (2005). Ownership and the internationalization of small firms. Journal of

Management, 31, 210-233.

Gioia D. A., & Chittipeddi K. (1991). Sensemaking and sensegiving in strategic change initiation. Strategic

Management Journal, 12, 433-448.

Godfrey P. C., & Hill C. W. L. (1995). The problem of unobservables in strategic management research. Stra-

tegic Management Journal, 16, 519-533.

Goodman R. (1980). Annual reports serving a dual marketing function—Report as survey. Public Relations

Quarterly, 36, 21-24.

Hair J. F., Anderson R. E., Tatham R. L., & Black W. C. (1998). Multivariate data analysis. Upper Saddle River,

NJ: Prentice Hall.

Hambrick D. C. (2007). The field of management’s devotion to theory: Too much of a good thing? Academy of

Management Journal, 50, 1346-1352.

Hart R. P. (2000). DICTION 5.0: The text-analysis program. Thousand Oaks, CA: Sage.

Hitt M. A., & Ireland R. D. (2005). The Blackwell encyclopedia of management: Entrepreneurship (2nd ed.).

Oxford, UK: Blackwell.

Holsti O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison-Wesley.

Hughes M., & Morgan R. E. (2007). Deconstructing the relationship between entrepreneurial orientation and busi-

ness performance at the embryonic stage of firm growth. Industrial Marketing Management, 36, 651-661.


Hult G. T. M., Snow C. C., & Kandemir D. (2003). The role of entrepreneurship in building cultural competi-

tiveness in different organizational types. Journal of Management, 29, 401-426.

Ireland R. D., Reutzel C. R., & Webb J. W. (2005). Entrepreneurship research in AMJ: What has been published,

and what might the future hold? Academy of Management Journal, 48, 556-564.

Ireland R. D., Webb J. W., & Coombs J. E. (2005). Theory and methodology in entrepreneurship research.

Research Methodology in Strategy and Management, 2, 111-141.

Jauch L. R., Osborn R. N., & Martin T. N. (1980). Structured content analysis of cases: Complementary method

for organizational research. Academy of Management Review, 5, 517-526.

Kabanoff B. Waldersee, R., & Cohen M. (1995). Espoused values and organizational change themes. Academy

of Management Journal, 38, 1075-1104.

Karren R. J., & Barringer M. W. (2002). A review and analysis of the policy-capturing methodology in orga-

nizational research: Guidelines for research and practice. Organizational Research Methods, 5, 337-361.

Kazanjian R. K. (1988). Relation of dominant problems to stages of growth in technology-based new ventures.

Academy of Management Review, 31, 257-279.

Kerlinger F. N., & Lee H. B. (2000). Foundations of behavioral research (4th ed.). Orlando, FL: Thomson

Learning.

Konrad A. M., & Mangel R. (2000). The impact of work-life programs on firm productivity. Strategic Manage-

ment Journal, 21, 1225-1237.

Kreiser P. M., Marino L. D., & Weaver K. M. (2002). Assessing the psychometric properties of the entrepre-

neurial orientation scale: A multi-country analysis. Entrepreneurship Theory and Practice, 26, 71-94.

Krippendorff K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA:

Sage.

Lance C. E., Butts M. M., & Michels L. C. (2006). The sources of four commonly reported cutoff criteria: What

did they really say? Organizational Research Methods, 9, 202-220.

Lumpkin G. T., & Dess G. G. (1996). Clarifying the entrepreneurial orientation construct and linking it to per-

formance. Academy of Management Review, 21, 135-172.

Lumpkin G. T., & Dess G. G. (2001). Linking two dimensions of entrepreneurial orientation to firm perfor-

mance: The moderating role of environment and industry life cycle. Journal of Business Venturing, 16,

429-451.

Lumpkin G. T., & Dess G. G. (2005). Entrepreneurial orientation. In M. A. Hitt & R. D. Ireland (Eds.), The

Blackwell encyclopedia of management: Entrepreneurship (pp. 104-107). Oxford, UK: Blackwell.

Lyon D. W., Lumpkin G. T., & Dess G. G. (2000). Enhancing entrepreneurial orientation research: Operatio-

nalizing and measuring a key strategic decision making process. Journal of Management, 26, 1055-1085.

Maitlis S., & Lawrence T. B. (2007). Triggers and enablers of sensegiving in organizations. Academy of Man-

agement Journal, 50, 57-84.

Marino K. E., Castaldi R. M., & Dollinger M. J. (1989). Content analysis in entrepreneurship research: The case

of initial public offerings. Entrepreneurship Theory and Practice, 14, 51-66.

Markel N. (1998). A semiotic contribution to libel action law. International Journal for the Semiotics of Law,

11, 49-56.

McAdams D. P., & Zeldow P. B. (1993). Construct validity and content analysis. Journal of Personality Assess-

ment, 61, 243-245.

McGahan A. M., & Porter M. E. (1997). How much does industry matter, really? Strategic Management Jour-

nal, 18, 15-30.

Mehran H. (1995). Executive compensation structure, ownership, and firm performance. Journal of Financial

Economics, 38, 163-184.

Meyer A. D. (1991). What is strategy’ s distinctive competence? Journal of Management, 17, 821-833.

Michalisin M. D. (2001). Validity of annual report assertions about innovativeness: An empirical investigation.

Journal of Business Research, 53, 151-161.

Miller D. (1983). The correlates of entrepreneurship in three types of firms. Management Science, 29, 770-791.

Morgan R. E., & Strong C. A. (2003). Business performance and dimensions of strategic orientation. Journal of

Business Research, 56, 163-176.

Morris R. (1994). Computerized content analysis in management research: A demonstration of advantages &

limitations. Journal of Management, 20, 903-931.


Mossholder K. W., Settoon R. P., Harris S. G., & Armenakis A. A. (1995). Measuring emotion in open-ended

survey responses: An application of textual data analysis. Journal of Management, 21, 335-355.

Naldi L., Nordquist M., Sjoberg K., & Wiklund J. (2007). Entrepreneurial orientation, risk taking, and perfor-

mance in family firms. Family Business Review, 20, 33-58.

Neuendorf K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.

Nunnally J. C., & Bernstein I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

Palmer T. B., & Short J. C. (2008). Mission statements in U.S. colleges of business: An empirical examination of

their content with linkages to configurations and performance. Academy of Management Learning and Edu-

cation, 7, 454-470.

Pearce J. A., & David F. (1987). Corporate mission statements: The bottom line. Academy of Management Exec-

utive, 1, 109-116.

Pennebaker J. W., Mehl M. R., & Niederhoffer K. G. (2003). Psychological aspects of natural language use: Our

words, our selves. Annual Review of Psychology, 54, 547-577.

Porac J. F., Wade J. B., & Pollock T. G. (1999). Industry categories and the politics of the comparable firm in

CEO compensation. Administrative Science Quarterly, 44, 112-144.

Potter W. J., & Levine-Donnerstein D. (1999). Rethinking validity and reliability in content analysis. Journal of

Applied Communication Research, 27, 258-284.

Rauch A., Wiklund J., Frese M., & Lumpkin G. T. (2004). Entrepreneurial orientation and business perfor-

mance: Cumulative empirical evidence. In S. A. Zahra, C. G. Brush, P. Davidsson, J. Fiet, P. G. Greene,

R. T. Harrison, M. Lerner, C. Mason, G. D. Meyer, J. Sohl & A. Zacharakis (Eds.), Frontiers of Entrepre-

neurship Research (pp. 164-177). Wellesley, MA: Babson.

Reilly F. K., & Wright D. J. (2002). Alternative small-cap stock benchmarks. Journal of Portfolio Management,

28, 82-95.

Riffe D., & Lacy S. & Fico, F. G. (2005). Analyzing media messages: Using quantitative content analysis in

research (2nd ed.). Mahwah, NJ: Erlbaum.

Rodale J. I. (1978). The synonym finder. Emmaus, PA: Rodale Press.

Rogelberg S. G., & Stanton J. M. (2007). Understanding and dealing with organizational survey nonresponse.

Organizational Research Methods, 10, 195-209.

Rosenberg S. D., Schnurr P. P., & Oxman T. E. (1990). Content analysis: A comparison of manual and com-

puterized systems. Journal of Personality Assessment, 54, 298-310.

Rumelt R. (1991). How much does industry matter? Strategic Management Journal, 13, 119-134.

Russell Investments. (2008a). Russell 30001 Index Fact Sheet. Retrieved March 23, 2009, from http://www.rus-

sell.com/indexes/characteristics_fact_sheets/us/Russell_3000_Index.asp

Russell Investments. (2008b). Russell 20001 Index Fact Sheet. Retrieved May 20, 2008, from http://www.rus-

sell.com/indexes/characteristics_fact_sheets/US/Russell_2000_Index.asp

Sarkar M. B., Echambadi R. A., & Harrison J. S. (2001). Alliance entrepreneurship and firm market perfor-

mance. Strategic Management Journal, 22, 701-711.

Savall H., Zardet V., Bonnet M., & Peron M. (2008). The emergence of implicit criteria actually used by

reviewers of qualitative research articles: Case of a European journal. Organizational Research Methods,

11, 510-540.

Scandura T. A., & Williams E. A. (2000). Research methodology in management: Current practices, trends, and

implications for future research. Academy of Management Journal, 43, 1248-1264.

Schnurr P. P., Rosenberg S. D., Oxman T. E., & Tucker G. J. (1986). A methodological note on content analysis:

Estimates of reliability. Journal of Personality Assessment, 50, 601-609.

Shane S. (2004). Encouraging university entrepreneurship? The effect of the Bayh-Dole Act on university

patenting in the United States. Journal of Business Venturing, 19, 127-151.

Short J. C., Ketchen D. J., & Palmer T. B. (2002). The role of sampling in strategic management research on

performance: A two-study analysis. Journal of Management, 28, 363-385.

Short J. C., Ketchen D. J., Palmer T. B., & Hult T. M. (2007). Firm, strategic group, and industry influences on

performance. Strategic Management Journal, 28, 147-167.

Short J. C., & Palmer T. B. (2003). Organizational performance referents: An empirical examination of their

content and influences. Organizational Behavior and Human Decision Processes, 90, 209-224.


Short J. C., & Palmer T. B. (2008). The application of DICTION to content analysis research in strategic man-

agement. Organizational Research Methods, 11, 727-752.

Singh K., Ang S. H., & Leong S. M. (2003). Increasing replication for knowledge accumulation in strategy

research. Journal of Management, 29, 533-549.

Skalski, P. D. (2002). Computer content analysis software. In K. A. Neuendorf (author), The content analysis

guidebook. Thousand Oaks, CA: Sage.

Slater S. F., & Narver J. C. (2000). The positive effect of a market orientation on business profitability: A

balanced replication. Journal of Business Research, 48, 69-73.

Smart D. T., & Conant J. S. (1994). Entrepreneurial orientation, distinctive marketing competencies and orga-

nizational performance. Journal of Applied Business Research, 10, 28-38.

Sonpar K., & Golden-Biddle K. (2008). Using content analysis to elaborate adolescent theories of organization.

Organizational Research Methods, 11, 795-814.

Stam W., & Elfring T. (2008). Entrepreneurial orientation and new venture performance: The moderating role of

intra- and extraindustry social capital. Academy of Management Journal, 51, 97-111.

Stevenson R. L. (2001). In praise of dumb clerks: Computer-assisted content analysis. In M. D. West (Ed.), The-

ory, method and practice in computer content analysis (pp. 3-12). Portsmouth, NH: Greenwood.

Tang J., Tang Z., Marino L. D., Zhang Y., & Li Q. (2008). Exploring an inverted u-shape relationship between

entrepreneurial orientation and performance in Chinese ventures. Entrepreneurship Theory and Practice, 30,

219-239.

Tsang E. W. K., & Kwan K. (1999). Replication and theory development in organizational science: A critical

realist perspective. Academy of Management Review, 24, 759-780.

Van De Ven A. H., & Johnson P. E. (2006). Knowledge for theory and practice. Academy of Management

Review, 31, 802-821.

Van Maanen J. (1979). Reclaiming qualitative methods for organizational research: A preface. Administrative

Science Quarterly, 24, 520-526.

Wade J. B., Porac J. F., & Pollock T. G. (1997). Worth, words, and the justification of executive pay. Journal of

Organizational Behavior, 18, 641-664.

Walter A., Auer M., & Ritter T. (2006). The impact of network capabilities and entrepreneurial orientation on

university spin-off performance. Journal of Business Venturing, 21, 541-567.

Weber R. P. (1990). Basic content analysis (2nd ed.). Thousand Oaks, CA: Sage.

Zahra S. A. (1991). Predictors and financial outcomes of corporate entrepreneurship: An exploratory study.

Journal of Business Venturing, 6, 259-285.

Zahra S. A. (1996). Governance, ownership, and corporate entrepreneurship: The moderating impact of industry

technological opportunities. Academy of Management Journal, 39, 1713-1735.

Zahra S. A., & Covin J. G. (1995). Contextual influences on the corporate entrepreneurship performance rela-

tionship: A longitudinal analysis. Journal of Business Venturing, 10, 43-58.


volume 13 number 2 construct validation using 2010 sage

Documents