iannaccone - reassessing church growth dani

8/3/2019 Iannaccone - Reassessing Church Growth Dani

1/21

Reassessing Church Growth: Statistical Pitfalls and Their ConsequencesAuthor(s): Laurence R. IannacconeSource: Journal for the Scientific Study of Religion, Vol. 35, No. 3 (Sep., 1996), pp. 197-216Published by: Blackwell Publishing on behalf of Society for the Scientific Study of ReligionStable URL: http://www.jstor.org/stable/1386549Accessed: 21/09/2008 15:41

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at

http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless

you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you

may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at

http://www.jstor.org/action/showPublisher?publisherCode=black.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed

page of such transmission.

JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with the

scholarly community to preserve their work and the materials they rely upon, and to build a common research platform thatpromotes the discovery and use of these resources. For more information about JSTOR, please contact [email protected].

Society for the Scientific Study of Religion andBlackwell Publishing are collaborating with JSTOR to digitize,

preserve and extend access toJournal for the Scientific Study of Religion.

http://www.jstor.org
http://www.jstor.org/stable/1386549?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/action/showPublisher?publisherCode=blackhttp://www.jstor.org/action/showPublisher?publisherCode=blackhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/1386549?origin=JSTOR-pdf


2/21

Reassessing Church Growth:Statistical Pitfalls and theirConsequencesLAURENCER. IANNACCONEt

Studies of churchgrowthhave fallen preyto numerousstatistical pitfalls. Flawed methods and inad-equate data have reduced he predictivepowerof past research while systematicallybiasingits results. Thebiases work to overstate the importanceof the demographiccontext in which a churchexists and under-state the importanceof the church'sown institutional characteristics,such as organizationalstrictness.This paperuses theory, simulations, and data reanalysis to explorethe empiricaldifficulties confrontingchurchgrowthresearchand to reassess the role of strictness.

INTRODUCTIONStudies of churchgrowth stretch back to the early part of this century,but the touchstoneof all moder research appearedin 1972 when Dean Kelley (1986) first published his land-mark book, Why Conservative Churches Are Growing. Kelley called attention to theunprecedenteddecline in the membershipof America'smainline Protestant denominations,which had begun around 1965, and he proposeda compellingbut highly controversialexpla-nation. Kelley argued that mainline denominations had becomeinsufficiently strict, shed-ding their distinctive demands and thereby losing their capacity to create meaning and togenerate commitment. The story seemed all the more persuasive because many strictdenominations (Mormons, Adventists, Jehovah's Witnesses, Southern Baptists, andAssemblies of God)were not in decline- indeed they were growing by leaps and bounds.In Kelley'sown words(1978: 165),the bookgeneratedconsiderable"controversy,f notactual scandal." Mainline denominational leaders rightly read it as a repudiation of thedirectionin which they had been movingtheir churches- the direction of greater individu-alism, ecumenical cooperation, social activism, and so forth. Critics accused Kelley of"deceptive statistics" (Bangs 1974: 852), "shallow" research (Bouma 1979: 136), anderroneous conclusions.1 Yet none could deny the reality of the mainline's membershipdeclines, nor the relative success of their more conservativecounterparts. Spurred by thesefacts, the mainline denominations and a bevy of mainline-affiliatedresearchersbegan study-ing church growth. By the normal standards of religious research, their efforts were well-funded and highly sophisticated, applying modern statistical techniques to mountains ofcensus and survey data collectedfromthousands of congregationsacross America'slargestProtestant denominations.

Laurence R. Iannaccone is a professor of economics at Santa Clara University, Santa Clara, CA 95053; e-mail:[email protected].

? -Johrnal for the Scientific Study of Religion, 1996, 35 (3J: 197-216 197


3/21

JOURNALFORTHESCIENTIFIC TUDYOFRELIGION

The fruits of this labor appeared in numerous books and articles but was most fullysummarized in two edited volumes published more than a decade apart: UnderstandingChurch Growth and Decline (Hoge and Roozen 1979a) and Church and DenominationalGrowth (Roozen and Hadaway 1993).2 Both volumes grew out of large research projects, andboth made heavy use of surveys commissioned by major mainline denominations, includingthe United Church of Christ, the United Methodist Church, and the United PresbyterianChurch. Both books garnered praise from noted religious scholars and church leaders.3Two conclusions stand out in this work. First, "contextual factors" (the demographicand socioeconomic characteristics of the communities surrounding a church) are at least asimportant, and generally more important, in determining church growth and decline thanare the "institutional factors" that a church might hope to control. Second, despite theimportance of some institutional factors (such as evangelism) and despite the fact that con-servative denominations continue to gain relative to their more liberal counterparts, Kelleywas wrong. Strictness does not influence or explain the growth. In an overview chapter tothe 1993 volume, editors Hadaway and Roozen (1993: 42) emphasize that "[r]esearch hasprovided little support for what has been called 'the Kelley thesis'. .. Strictness ... is unre-lated to growth within liberal or conservative families. Kelley's 'theory' is best understood asone of sectarian survival - not congregational or denominational growth." Authors repeat-edly express this view throughout both volumes, and its status as demonstrated fact mayexplain why only one study in the first volume (Hoge 1979) and none in the second wereexplicitly designed to test the strictness thesis.4These two conclusions may have given some solace to the shrinking mainline. Thefirst suggests that they really cannot do much (nor could they have done much) to stave offdecline. The second asserts that insofar as they can do something, it need not be anything soradical as a turn toward the strictness, separatism, and other-worldliness that characterizesthe conservative wing of American Protestantism.A review of the data and methods underpinning most church growth research castsserious doubt on the basis for both conclusions. Statistical problems abound in past studies,problems that have gone largely unexamined.5 These not only reduce the predictive power ofthe research; they systematically bias its results. Nearly all the biases work to overstate theimportance of demographic context and understate the importance of a church's own institu-tional characteristics, most notably its organizational strictness. This paper uses theory,simulations, and the reanalysis of old data to identify the statistical problems in pastresearch. My basic message is summed up by a methodological warning found in Hadaway(1989: 160): "If the researcher knows where the mines lay and how to avoid them, then goodresearch is possible, but for the unwary . . . the chances for fatal errors are very high."The paper has obvious relevance for churches that seek to grow, but it also relates to adeveloping body of theory and data that models religious organizations in terms of the coststhey impose on their members (Stark and Bainbridge 1985, 1987, Finke and Stark 1992,Iannaccone 1992, 1994). According to this work, strictness promotes strength by discourag-ing free riding and screening out less committed members. The rehabilitation of Kelley'sthesis gains wider significance when viewed against the backdrop of work that embedsstrictness within a broad theory of sectarianism, rational action, and the evolution of reli-gious organizations.

UNDERSTANDING THE PITFALLS

The pitfalls I address are all instances of standard statistical problems. One maytherefore turn to any intermediate statistics text to see the general problems analyzed inabstract, mathematical terms. For present purposes, however, equations are less illuminat-ing and less compelling than concrete examples. Moreover, since the relationships behind

198


4/21

REASSESSINGCHURCHGROWTH:TATISTICAL ITFALLSAND THEIRCONSEQUENCES

any real-worlddata set remain foreveruncertain, I will illustrate each of the pitfalls withstatistical simulations. The simulations can be fully specified and repeated at will, leavingno doubt as to their underlyingstructure and statistical properties.Thoughone may debatethe extent to which any particular simulation approximates real-world situations, I seekmerely to show that widely used methods lead to incorrect inferences when applied torepresentative data. The plan thereforeis to take popularmethods, applythem to simulateddata structured to mimic the theories that researchers have proposed, and show how themethods break down.6 Having identified problems in this fashion, I will then suggestremedialactions that can avoid ormitigate them when workingwith real data.To keep the analysis as concreteand realistic as possible, I will adopt the conceptualframeworkthat has dominatedprevious research. Membership growth will be attributed to"contextual"and "institutional" actors, and denominations will vary along a continuum ofstrictness/conservatism.Pitfall 1: Restricting the sample (to a single denomination)understates the impact ofinstitutional characteristics

When scholars speak of Kelley'sthesis having been refuted, they usually are referringto large-scale, census and survey-basedstatistical studies of congregationalgrowth. Nearlyall these studies looked for the statistical correlates of church growth within a singledenomination.So, forexample, Hadaway(1980)and Roof,Hoge, Dyble, and Hadaway(1979)identified the correlates of churchgrowthin a sample of 681 United Presbyteriancongrega-tions; McKinney(1979, cf. McKinneyand Hoge 1983) undertook a similar analysis of 263United Church of Christ congregations;Perry and Hoge (1981) studied 205 Presbyteriancongregations; and Thompson, Carroll, and Hoge (1993) studied yet another 593Presbyterian congregations. All these studies paid some attention to strictness or conser-vatism, and nearly all concluded that strictness had but "weak"effects on congregationalgrowth. Moreover, "institutional" variables of every sort tended to look weak relative to"contextual"variables such as population growth, neighborhood characteristics, incomes,and the like.Thereis, however, an overwhelmingproblemwith these studies. A denomination is byits very nature committed to a particularset of institutional characteristics - theological,behavioral, and organizational. Hence, the congregations within a given denominationinevitably manifest much less institutional variation than that which actually exists acrossthe entire religious market. This narrowness is likely to be especially great with regard tothe attributes that constitute strictness. No Mormoncongregation,howeverliberal, is free toignore LDS church rules regarding tobacco, alcohol, tithing, temple rites, organizationalstructure, excommunication, and the like. And conversely, no Presbyterian congregation,howeverconservative,is in a positionto impose these rules. On the other hand, most contex-tual/demographicattributes vary greatly across the congregationsof a single denomination.The average Presbyterian may be relatively well off, but some congregationsare poor andless educated,have many large families orhigh rates of unemployment,are locatedin ruralareas or areas of increasing population,and so forth.In short, looking across the "observations"of a single-denomination data set, thecontextual variables always exhibit much more variation (relative to that of society as awhole) than do the institutional variables. The situation parallels that of an agriculturalexperiment in which rainfall levels are nearly identical on every test plot, but fertilizerlevels vary substantially. Such an experiment will inevitably find that actual variation incrop yield dependsmore on the observedvariationin fertilizer than the observedvariationinwater (whatever the actual importanceof water). Single-denominationstudies are likewisebound to find that institutional variables (particularly those most tightly associated with

199


5/21

JOURNALFORTHESCIENTIFIC TUDYOF RELIGION

denominationaldifferences)accountfor a relatively low fraction of the observed variation ingrowth.A simulation highlights the problem.Assume that congregationalgrowth is, in fact,the sum of three independent factors: a contextual variable (growthin the local population),an institutional variable (strictness), and unobservable effects. Hence, the true regressionequationforgrowthis

GROWTH POP _ GROWTH STRICTNESS+ e (1)

Let each observable variable account for 25%of the variation in membership growth so thatthe true R2is .50. (To keep things simple, let all the independent variables be normallydistributed across the full population of Protestant congregations.This assumption is notnecessary, but it guarantees the applicability of standard regression statistics.) Assume,moreover,that each congregation belongs to one of eight denominations arrayed along thestrictness continuum.Within each denomination,congregationsare drawnrandomlyfrom astrictness "band" hat covers a limited (1.5 standard deviation) portion of the entire strict-ness distribution.The bands of adjacentdenominationsoverlap,so that the congregationsofthe first denominationare not all less strict than the congregationsof the second denomina-tion, and so forth. On the other hand, assume that local population growth rates varyindependentlyof denomination.

FIGURE 1GROWTHVERSUS STRICTNESS ACROSS A RANDOM SAMPLE OF SIMULATEDCONGREGATIONS

* . .~~~~5- *..-. ..

U -. ? . . '

? .o.%..?....

. . -

0 i 2 3 4 5 6 7Strictness

.&Io" '7-'" .

StrictnessNote: Circlesdepictthe congregationsof a single denomination.Dots depictthe congregationsof all other denomina-tions.

200


6/21

REASSESSINGCHURCHGROWTH: TATISTICAL ITFALLSAND THEIRCONSEQUENCES

To simulate this situation using standard statistical software, one may randomlygenerate numerical values for each simulated congregation's POP_ GROWTH,TRICTNESS,nde; and then calculate the congregation'sGROWTHccordingto equation (1). Figure 1 showsthe results from one such simulation, computedwith the Stata statistics package.The figuredisplays the relationship between growth and strictness for a thousand randomlygenerated"congregations."The stipulated positive relationship between growthand strictness is quitevisible. Applyingstandard regression techniques to a randomsample of 100 congregations,one obtains the coefficients,t-statistics, and R2foundin columns 1 through3 of Table 1.Note that the regressions recover the "right" esults from the simulated data, whichis to say results consistent with the actual, underlying model specified by equation (1).Working from a random sample of all the congregations, we correctly infer thatPOP_GROWTHnd STRICTNESSare each significant, equally important, and each accountforabout a fourthof the variance in GROWTH.

But what if we restrict the analysis to a single denomination?The circles in Figure 1correspond to the congregations of just one fairly liberal denomination. Following theassumptions of the simulation, these congregationsall fall within a fairlynarrowsegment ofthe strictness continuum. Within this restrictedsample, the actual, underlyingreiationshipbetween growth and strictness is far from clear. On the other hand, the effect of populationgrowth remains strong since the denomination'scongregations vary widely with respect tothis "contextual" ariable.TABLE 1

GROWTHREGRESSIONS IN CROSS-DENOMINATIONAND SINGLE-DENOMINATIONDATA(1) (2) (3) (4) (5) (6)

POP__GROWTH 1.05*** 0.96*** 0.57*** 0.58***(6.8) (7.4) (5.0) (5.1)STRICTNESS 1.03*** 0.93*** 0.03 0.22(5.8) (6.5) (0.1) (0.6)CONSTANT 0.38 -2.58 -2.40 -0.96 -1.26 -1.46R2 .32 .26 .52 .21 .00 .21Observations 100 100 100 100 100 100Note:Asterisks *, **,and ***denote significanceat the .05 and .01, and .001 levels, respectively. Dependentvariableis congregationgrowth.Regressioncoefficients are in plain type, t-statistics in parentheses.Data are simulated accord-ing to equation(1) in the text. Columns1 through3 regressover a randomsampleof congregations rom all denomina-tions;columns4 through6 regressovercongregations romone denomination.

Regressions over 100 simulated congregations drawn randomly from this singledenominationyield the results printed in columns 4 and 6 of Table 1. In these regressions,the estimates for POP_GROWTHemain approximatelycorrect,but the R2 and t-statistics forSTRICTNESSare far too low.7Whereas population growth still explains around one fourth ofthe variance in growth, strictness now accounts for virtually nothing. Faced with theseresults, a researcher would certainly concludethat institutional effects are "weak" elativeto contextual effects. But the weakness is an illusion, an artifactof the restricted,denomina-tional sample. The true relationship remains that of equation(1).Although the specificnumbers in Table 1 reflect the assumptions built into the simu-lation, the basic results remain valid for virtually any conceivablereal-worldsituation. Onethus finds that any study of congregationswithin a single denomination nevitably underes-timates the relative importance of institutional variables such as strictness, doctrine,organizationalstructure, and the like.8 Moreover,the problempersists even in multidenom-

201


7/21

JOURNALFORTHE SCIENTIFICSTUDYOF RELIGION

inational studies as long as the denominations occupynearby points on the denominationalspectrum.A randomsample of congregationsfromthe Episcopal,United Churchof Christ,United Presbyterian,and United Methodist churches,still represents a small fraction of thestrictness spectrum that extends to Adventists, Mormons, Jehovah's Witnesses, andbeyond.9This fact validates the concernvoicedyears ago by Dean Kelley (1979:335):

Our authors[ofchurchgrowthresearch]... eschew the effort to obtain data on the most intense formsofreligiousbehavior 'newreligiousmovements, sects, and cults' - and confine themselves to the culture-affirmingmainline churches... [A]whole wing of the spectrumof religiousbehavior n this country byfar the most interestingand informative has been eftout. [Italicsoriginal]

Pitfall 2: Searchingovernumerous variables inflates R2and invalidates significancetestsSignificance tests were designed for simpler times. Decades ago, an agricultural

researcher might seed a hundred plots of land, varying the level of fertilizer from one toanother. He would then hand calculate a single bivariate regression to test whetherfertilizerhad a "statistically significant"effect. If the estimated coefficient was significantlygreater than zero at the 5% evel, he might reasonably infer that fertilizer does affect cropyield, since the odds of getting such a strong result purely by chance was less than one intwenty.The appropriate nferences change dramatically if the researcherhad not only testedfor the effects of fertilizerbut had also measured and calculatedyield regressions for each of99 other factors - soil composition, drainage, rainfall, shade, wind, and so forth. With atotal of 100 explanatory variables, standard t-tests can be expected to turn up five"statistically significant"effects even if none of the variables exerts any real influences oncrop yield. This, after all, is what a "5 percent significance level" means - a statistic soextreme that mere chancewouldcause it to arise only five times out of a hundred.With fiveor perhaps even ten "significant"correlations out of a hundred, the researcher has nobusiness inferringanything fromthe data.Sadly, the methods of most large-scale congregationalstudies exactly parallel those ofour hapless farmer. Working with hundreds of hypothetical explanatory variables, theytypically correlateeach variable with growth and then retain those that prove "significant"at the 5%, 10%,or 15% evel. But the significance test in question applies only to a singlecomparison involving a single, preselected variable! Is it any wonder that these studiesroutinely identify 20 to 30 "significantpredictors"of growth?Mere chance would yield 15spurious predictors n a data set with 300 purely random variables.This criticism might sound harsh, but consider several authors' own accounts of theirprocedures n recent studies:

Because of the extensiveness of the data (more then 700 variables)a series of preliminaryanalyses wasconducted to reduce the sets of predictor variables to a more manageable number that could beincorporatedinto the final analyses .... All potential predictorvariables that displayed statisticallysignificant (at p < .05 level) and theoretically meaningful correlationswere retained and groupedintoseveralsubsets. (Welch1993:327, 370)The length and complexity of data displays discussed thus far precludecogent summary. Such mind-numbing'laundrylists' of variables ... becloudlarger issues concerningwhich characteristics are moststronglyrelatedto congregational rowthand decline.... The table displays only those predictors hat hadpartial R values of .10 or higher, after controllingfor all of the other characteristicsin that theme ....Havingthus sifted throughthe variables ... a 'grandmodel'was constructed. Donahueand Benson 1993:235-37)Over200 predictorvariables wereavailable for the analysis. Wecorrelatedall of them with the member-ship changemeasure and retainedonly those havingcorrelationssignificantat or near the .10 significancelevel ... A total of 32 variableswere retained.(Thompson,Carroll,and Hoge1993:192)

202


8/21


Such procedures cannot help but identify "significant"variables that are, in fact, totallyirrelevant. A Nobel laureate, RonaldCoase, once put the matter bluntly:"ifyou torture thedata long enough, Nature will confess"(Kennedy 1985: 76). These false confessions growindirectproportion o the number of variables and/ormodels over which one searches. Hence,searching for the significant predictors among a list of several regressors may not be toomisleading (and is in fact common),but searching over several hundred predictors almostalways yields nonsense. When the quoted authors identify 32 explanatory factors(significant at the .10 level) from an initial set of 200, one must recall that 20 such factorswould arise in a purelyrandom set of data.The errors multiply when the researcher proceeds from searching to regressing,testing groups of "significant"variables for their contribution to R2. The resulting regres-sions inevitably explain a substantial portion of their variance, even if the underlyingvariablesarepure noise.The problem s multicollinearityand insufficientdegrees of freedommaskedby the initial search.In fairness to the church growth literature, most researchers do make a point ofadjusting their R2calculations to account for degrees of freedom. For example, Thompson,Carroll, and Hoge (1993: 207) report that "All the R2's are adjusted downward using thestandardadjustmentformula :. . based on the number of cases and the numbers ofvariablesin regression."Theitalics are mine.) McKinneyand Hoge(1983:60), Welch(1993:371), Roofet al. (1979: 368), and most other previously cited studies likewise adjust their regressionsforthe numberof explanatoryvariables in the regression.But the authorsoverlookthat meaningful adjustmentsmust take account of the entireset of variables that were examined, not just the few that happen to have been retained.Regressing growthonto the 10 "significant"variables in a 50-variable data set is tantamountto regressing onto the entire set of 50 variables. After all, the 40 "insignificant"variablesadd little or nothing to R2 once we have already included the 10 "significant"variables. Itfollowsthat in such cases the appropriate adjustment to R2 comes not fromthe 10 variablesthat were retainedbut rather from the entire 50 that were examined.The numerical consequences are immense. The standard adjustment isadj-R2 = 1-( 1-R2)(n- 1/(n-k-1 )), where "n"denotes the number of observations and "k"the number of regressors. This inflates the unexplained portionof the variance by(n-1)/(n-k-1).A 100-observation,50-variableregression with a rawR2 of .50 thus convertstoan adjustedR2 of zero.But consider the actual studies. Roof et al. (1979: 199) derive their regressions from"681congregations .. . [and]nearly 500 variables."This implies that any raw R2under .73adjusts downto zero.McKinney 1979: 233) works with 263 congregationsand 70 predictors,implying that any R2 below .27 may be entirely spurious. Thompson,Carroll,Hoge (1993:189, 192)work with 593 congregationsand "[o]ver200 predictorvariables," mplyingthat wediscount any R2 under .34. Similar considerations apply to all the previously cited studies.Sadly, the actual R2obtained in most of these studies falls below these levels. Roof et al.obtain an R2 of .26;McKinneyreports an R2 of .25, and Thompson,Carroll,and Hoge'smostcomprehensiveregressionhas an R2of only .32.The readerwill note that the cited studies never actually ran their regressions over allavailable predictorsbut instead built their regressors from the strongest zero-ordercorre-lates of growth. This makes my preceding readjustments too severe. But simulationsdetailed in my technical appendix show that the two-step search and regress procedureremains deeply flawed in any case (yielding long lists of spurious predictorsand R2 valuesthat overstate equations' predictivepower).10The only solution is restraint. To producevalid test statistics, the researcher must notwork with hundreds of explanatory variables, must swear off searching, and must limitregressions to a few well-defined, predetermined variables. Kitchen-sink correlations may

203


9/21

JOURNALFORTHESCIENTIFIC TUDYOFRELIGION

have use in purely exploratory work (though they inevitably turn up more junk than truth),and they may help show that a particular preselected theory is truly robust (by proving thatits variables remain significant even when one tries drowning them with controls), but theymust not be confused with hypothesis testing.Pitfall 3: Measurement errors (that affect institutional variables more than contextualvariables)understate the importanceof institutions

Applied researchers have a hard life. Even when their models are correctly specified,and even when they have identified the true causes and have excluded the spurious corre-lates, they must struggle with measurement error. Josiah Stamp (1929: 258-9) summarizedthe dilemma long ago:

The Governmentarevery keen on amassingstatistics - they collectthem,addthem, raise them to the nthpower,take the cube rootand preparewonderfuldiagrams.But what you must never forgetis that everyone of those figures comes in the first instance from the village watchman,whojust puts down what hedamnpleases.Substitute "denominational headquarters" for "government," and "congregational secretary"for "village watchman," and you have the problem confronting church researchers.Statistical horror stories abound and need no repetition. It suffices simply to note thatmost congregations maintain sloppy records, have few incentives for getting their numbersright, operate with little or no oversight, and face no penalties for error. Church membersare likewise prone to misstate facts regarding, for example, their rates of church attendanceor contributions, through thoughtlessness, forgetfulness, miscommunication, or outrightdeceit. A still greater margin for error and miscommunication exists when polling peopleabout their personal feelings, beliefs, and attitudes toward their congregations.Church researchers must live with these problems, but they must also acknowledgethe biases they introduce. When researchers test a theory about factor X (e.g., "strictness")but have at their disposal only some loose proxy for X, then the estimated impact of X will bebiased toward zero (Kennedy 1985: 113, 123). Moreover, if the researchers are seeking toassess the effects of two different sets of factors X and Y, and random measurement errorsplague X more than Y, then the regression results will be systematically biased in favor of Y.The estimated coefficients, betas, t-statistics, and R2 will all suggest that Y has more impactthan X, even if both are equally important.Now it would seem that the "contextual" variables in most church growth models aremuch less error-filled than the "institutional" variables, since the former are demographicquantities derived from systematic, large-sample government census statistics (of populationgrowth, property values, incomes, family size, racial distributions, and so forth), whereas thelatter are mostly attitude scales derived from relatively small, nonrandom samples ofcongregation's membership.11 Measures of "strictness" may be particularly error-prone,because very few mainline congregational surveys even mention the inventory of activitiesand attitudes that Kelley (1986: 84) equated with strictness. This means that most tests ofdemographics versus strictness, or more generally, external context versus internalinstitutions, will overstate the relative impact of the former.

To see the bias at work, assume that membership growth is, in fact, equally deter-mined by population growth and strictness [as was previously assumed in equation (1)]. InTable 1, columns 1 through 3, we saw that regression yields correct conclusions whenworking with the true variables and a random sample of simulated congregations. Assumenow that the researcher cannot observe true population growth and strictness but mustinstead use proxy variables. Assume moreover that the population proxy is much less

204


10/21


"noisy"han the strictness proxy. Specifically,letPOP-PROXY= POP_GROWTH ?1

STRICT_PROXY STRICTNESS+ ?2where all the right-hand side variables have independent normal distributions, with thevariance in 61 one tenth that of POP_GROWTHand the variance of 62 equal to that ofSTRICTNESS.Regressing GROWTH nto the true variables and their proxies (in a 1000-observation,simulated data set) then yields the results printedin Table 2.12

TABLE 2GROWTHREGRESSIONSWITH ERRORS IN THE VARIABLES

(1) (2) (3) (4) (5) (6)POP_GROWTH .48 .48(17.39) (20.65)STRICTNESS .48 .48

(17.25) (20.52)POP_GROWTHPROXY .45 .46(15.98) (16.97)STRICTNESSPROXY .26 .24(7.16) (8.86)

R2 .23 .23 .46 .20 .05 .26Observations 1000 1000 1000 1000 1000 1000Note: Beta coefficients in plain text; t-statistics in parentheses. Data are stimulated via growth = pop_growth +strictness + error.Proxyvariabledefinitionsare in the text.

Columns 1 through 3 show the results obtained when GROWTHs regressed onto itstrue determinants, POP_GROWTHand STRICTNESS, ither singly or jointly. The resultscorrectly mply that both factorsare equally important, each accountingforaboutone fourthof the total variancein growth.Columns 4 through 6 show the corresponding regression of growth onto the proxyvariables for population growth and strictness. Note that these results seem to show thatdemographicsare far more significant than strictness - the populationproxyexplains 20%of the variance in growth, whereas the strictness proxy explains only 5%;the populationproxybeta coefficients are nearly twice the strictness betas; and the populationt-statisticsare nearly twice the strictness t-statistics.13 But the apparentlyobviousconclusion s wrong.We know that both factors are equally important and both contribute 25% because wesimulated the data using equation (1). Strictness appears less important only because itsmeasurement involved a relatively large error.In a case like this, the only practicalsolutionis a majoreffort to obtain better measures of strictness, something largely absent from thegrowthliterature.Pitfall 4: Stepwise-hierarchicalregressionsthat force institutional effectsto follow contextualeffectsare biased

Church growth studies typically assess the relative importance of contextual versusinstitutional factors by means of stepwise-hierarchical regressions that first calculate theexplanatory power of contextual variables and then estimate the additional explanatorypower derived from institutional variables. For example, Thompson, Carroll, and Hoge

205


11/21

JOURNALFORTHE SCIENTIFIC TUDYOFRELIGION

(1993: 199) divide their explanatoryvariables into three groups that they call "communitycontext,""congregationaldemography,"and "institutional factors"(which include recruit-ment efforts, theological orientation, church facilities, organizational structure, congrega-tional attitudes, and the like). Regressing growth onto the first set of factors yields anadjusted R2 of .16; adding the second set of factorsincreases R2 to .20; and adding the finalset increases R2 to .32. From these numbers the authors infer that "communitycontextaccountsfor 16%of the variance, congregationaldemographyaccounts for an additional4%,and institutional factors account for another 12%."Taking the same approach,Roof et al.(1979: 220) report that contextual variables account for 14.6%of the growth in churchmembershipand that the additionof institutional variables accounts for another 11.4%.Theoverall conclusion?Thompson,Carroll,and Hoge (1993: 205) observe that in both studies"contextualfactors were a bit stronger than institutional factors." (Similar proceduresandresults characterizeHoge 1979, McKinney1979,McKinneyand Hoge 1983, Olson 1993, andWelch 1993.)There is good reason to question this conclusion, if only because it follows a tableindicating that institutional factors are substantially more important than communitycontext (Thompson,Carroll, and Hoge 1993: 193-96). The table groups all 32 explanatoryvariables into clusters, seven of them contextual and seven institutional, and lists thepercent of variance in growthindependentlyexplained by each cluster (i.e., the adjusted-R2when growth is regressed onto all the variables in that cluster). The highest contextualcluster explains 11% followedby other contextualclusters that explain 8, 7, 4, 4, 2, and 1%),whereas the highest institutional cluster explains 14%(followed by other institutionalclusters that explain 12, 11, 6, 6, 5, and 4%).Whyconcludethat contextual factorsare moreimportantthan institutional factors,when each institutional cluster explains more variancethan the correspondingcontextual cluster?And why say that all the institutional variablestogether account for only 12%of the variance in church growth, when one of seven institu-tional clusters predicts 14%of the varianceall by itself?As it turns out, the authors' conclusionsderive entirely fromthe predeterminedorderin which variables entered the regressions: first context, then institutions. Very differentresults wouldhave emergedhad the orderof entry been reversed. As things stand, anythingthat can be (statistically) attributedto context is attributed to context; institutional factorsonly get a shot at what is left over. To be sure, Roof et al. (1979: 200) do acknowledgethat"the size of the coefficients can be influenced by the order in which the clusters areintroduced" see also McKinneyand Hoge (1983:63) and Welch (1993: 371). But this criti-cal proviso is glossed over on the grounds that the order of inclusion parallels the "causalorder"of the variables. The researchers assume that in any statistical model demographicvariablesmust precedecongregationalattitudes because the latter are not likely to influencethe former,whereas the formermight very well shape the latter.The assumption is wrong.Even if contextualfactors are causally prior to institutionalfactors, the stepwise-hierarchicalapproach yields biased results that systematicallyunderes-timate the importanceof institutions.14To illustrate the fallacy, considera study of how mold and excessive rain inhibit thegrowth of grapes. Rain can damage grapes directly (by causing them to split) or indirectly(by stimulating the growth of mold).Rain is thus "causallyprior" o mold. Does it thereforemake sense to regress crop yields onto rainfall (obtaining an R2 of, let us say, .80), thenexpand the model to include the level of mold (obtaining an R2 of .90), and then concludethat "moldexplains only 10%of the variance in growth"or that "rainis eight times moreimportant than mold when it comes to determining grape yields"? By no means. Thestepwise strategy fails on two counts: first, by describing the situation so as to understatethe impact of mold (which might well account for most crop losses, while grape-splittingaccounts for a little); and second,by directing attention away frommold-orientedsolutions.

206


12/21


Farmers are led to focus on rain, a factor beyond their control, rather than inexpensivefungicides like sulfur. In the same manner, a stepwise approachto church growth leads tomisleading descriptionsand biased prescriptions.Simulations highlight the problemand suggest a solution. Considerthe various pathmodels of church growth depicted in Figure 2. Arrowsdepict causation, dotted lines depictnoncausal correlation. The figures seem to encompass all the majorscenarios that scholarshave suggested. In the first, incomeand strictness affect growth equally and independently.In the second, income and strictness are correlated,but only income influences growth. Inthe third, the situation is reversed and only strictness influences growth. In the fourth,income and strictness are correlated(due, perhaps, to some commonprior factor)and bothinfluence growth. In the fifth, income affects strictness, which in turn affects growth, butincomeitself has no direct impact upon growth.The sixth case parallels the grape example,since income now affects growthboth directlyand indirectly.15With a few additional assumptions (such as the assumption that all observable andunobservablefactors are normallydistributed and the assumptionthat each true cause con-tributes 33%to the actual variance in growth)one may readily simulate all six models. Forexample, to simulate the first model, I generated three independent and identically dis-tributed random variables; labeled these variables INCOME, STRICTNESS, and e; and thencalculated GROWTHaccordingto the equation GROWTH= (-1)-INCOME + (+1).STRICTNESS + ?.To simulate the second model, I generated the independent variables in such a way that

INCOME correlates with STRICTNESS, then calculated GROWTH according to the equationGROWTH (-1)INCOME + E. Proceedingin this manner, I producedsix sets of simulated datacorrespondingto the six models in Figure 2. Each data set was then analyzed using thesame stepwise-hierarchicalregressionmethodscharacteristic of past research.Table 3 shows the results. In every case except the first two, the stepwise-hierarchicalapproach overstates the influence of income, the contextual factor, and understates theinfluence of strictness, the institutional factor. Consider, for example, the third columnwhich summarizes results for model 3. The data for model 3 were simulated in such a waythat strictness and income are correlated but only strictness affects growth. Hence, as isnoted in the "actualR2"rows, incomecontributesnothing to this model'sactual variation ingrowth, whereas strictness contributes .33. But looking at the "estimated R2"numbers, wesee that regressing growth onto income alone yields an estimated R2 of .15, and regressinggrowthonto income and strictness yields an estimated R2 of .30. This leads to the erroneousconclusion, isted under"imputedR2," hat incomeand strictness are equally important.Theerrorhas a simple cause: in this model,and every model except the first two, the underlyingcorrelationbetween income and strictness guarantees that the initial (context-only) regres-sion suffers from a classic "omitted variable" problem that inflates R2 and biases theestimated coefficient. To justify the stepwise approach, a researcher must thus posit anabsence of correlationbetween contextual and institutional variables (which clearly is false)or the impossibilityof a true casual effectrunningfrom the institutional factors to growth-the very hypothesis that the approachwas supposedto test!16Reversing the order of stepwise entry will not solve the problem, though it mayhighlight the problem'sexistence. The only real alternative lies in reporting the completeresults for a multiple regression in which both factors appear simultaneously. Correlationamongthe independent variables will still make it difficult to portionout R2;but comparingthe magnitudes of t-statistics and/or betas in a single, multiple regression is much moredefensible than comparingthe increments to R2in a series of hierarchicalregressions. Table4 reportsthe results of standard multiple regressionsusing the same data. In every case theresults are consistent with the actual underlyingmodelused to generate the data.

207


13/21

JOURNALFORTHESCIENTIFICSTUDYOF RELIGION

FIGURE 2PROPOSED MODELS OF GROWTHAS A FUNCTION OF INCOMEAND STRICTNESS

Model 1: Income and strictnessindependentlynfluencegrowth.

Model 2: Incomeand strictness arecorrelated,butonlyincome influencesgrowth.

Model 3: Income and strictness arecorrelated,butonlystrictnessinfluencesgrowth.

Model4: Incomeand strictness arecorrelated,and each influencesgrowth.

Model 5: Income influencesstrictness,and strictness influencesgrowth.

Model 6: Income influencesstrictness,and bothinfluencegrowth.

incomegrowth

strictnessincome

t^%^' growth' - strictness

incomegrowth

--. strictnessincome,

It ' ~ ^ growth- -strictness

incomegrowth

strictnessincome

inc ^^^^-^growthstrictness

208


14/21

REASSESSINGCHURCHGROWTH:TATISTICAL ITFALLSAND THEIRCONSEQUENCESTABLES

STEPWISE-HIERARCHICALREGRESSION RESULTS(ACTUALAND IMPUTED CONTRIBUTIONSTO VARIANCEIN GROWTH)

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6Estimated R2:INCOME nly .32 .30 .15 .52 .24 .61INCOME+ STRICTNESS .65 .30 .30 .63 .32 .66ImputedR2:to INCOME .32 .30 .15 .52 .24 .61to STRICTNESS .33 .00 .15 .11 .08 .05Actual R2:from INCOME .33 .33 .00 .33+ .00 .33+

fromSTRICTNESS .33 .00 .33 .33+ .33 .33+Observations 1000 1000 1000 1000 1000 1000Note: Each modelis simulated according o the correspondingpath diagramin Figure 2. A ".33+"denotes models inwhich incomeand strictness arecorrelated,contributeequallyto growth,andjointly account or66%of the variance ngrowth."EstimatedR2"rows list R2 forregressionsof growthontoincome alone and then onto income and strictnesscombined."ImputedR2" ndicates how the regressionresults areinterpretedfollowinga stepwise-hierarchical cheme."ActualR2"gives the true structureof the simulated data.

TABLE 4SIMPLE VERSUS MULTIPLE REGRESSIONS FOR GROWTH

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6Simple Regression:INCOME nly .56 .55 .39 .72 .49 .78(21.43) (20.63) (13.22) (32.67) (17.93) (39.50)R2 .32 .30 .15 .52 .24 .61Simple Regression:STRICTNESSonly .57 .39 .55 .73 .32 .77

(22.11) (13.22) (20.63) (34.00) (21.64) (38.32)R2 .33 .15 .30 .54 .32 .59............. ............. ....................Multiple Regression:INCOME .56 .57 -.03 .41 .07 .45(29.80) (14.63) (8.82) (15.65) (1.46) (13.36)STRICTNESS .57 -.03 .57 .45 .51 .39

(30.44) (0.82) (14.63) (17.27) (10.64) (11.62)R2 .65 .30 .30 .63 .32 .66Observations 1000 1000 1000 1000 1000 1000Note: Beta coefficients n plain text; t-statistics in parentheses.Same data as in Table 2, simulated fromthe modelsdepicted n figure2.

The moral is clear. Stepwise-hierarchical regressions that force institutional effects tofollow contextual effects are biased and must be avoided. The conclusions drawn from suchstatistics cannot be trusted.

209


15/21

JOURNALFORTHE SCIENTIFICSTUDYOF RELIGION

ESCAPINGITFALLS:TRICTNESSEVISITEDIt is by now clear that potentially serious problems plague past studies of church

growth. To demonstrate that the problemshave, in fact, led to inappropriate conclusions,and to see how they can be overcome,I will reanalyzethe data from a study entitled "ATestof Theories of Denominational Growth and Decline." I have chosen this study for severalreasons. It is one of the few studies that work with denominationaldata, thereby avoidingthe problemsof a single-denomination sample. It is the only study designed from the groundup to test Kelley's theory of strictness and growth.As such, the study's author, Dean Hoge,went to great lengths to measure the social, organizational, and doctrinal attributes thatKelley identified as the determinants of churchstrength. Most importantly, the author doc-umented his work with such care that one may fully recover his data and fully replicate hisanalysis.17 In the years since Kelley first wrote about strictness, no empirical study hasaddressed his thesis with anything approachingthe precision,ingenuity, and thoroughnessof Hoge'stest.Since Hoge described and documented his work so well, it suffices merely tosummarize its major features. First, the study worked with denominations as the unit ofanalysis. This limited the data to 16 cases corresponding o the majorProtestant denomina-tions for which a wide range of data were available. Second, the study sought to explaindenominationalgrowthrates in terms of other denominationalattributes, both "contextual"and "institutional." Hoge derived his contextual measures (including average income,education, occupational prestige, family size, and regional distribution)for each denomina-tion fromgovernment data, the Glenmarycensus of religious bodies, and NORC'sGeneralSocial Surveys. In order to obtain institutional measures correspondingto Kelley's theory,Hoge asked 25 experts (church historians, sociologists of religion, and denominationalleaders) to completea questionnairein which they ranked each of the denominationsalong aseries of seven-pointscales. The scales concerned he denominations'strength of ethnic iden-tity, theological conservatism, attitudes toward ecumenism, centralized or congregationalpolity, emphasis on local and community evangelism, involvementin social action,emphasison distinctive life-style and morality,and attitude towardpluralism of beliefs.Giventhe relatively large numberof explanatoryvariablesand the very small numberof cases, searching for significant correlations is problematic.Fortunately, all the institu-tional ratings (except"ethnic dentity"and "polity")were so stronglyintercorrelatedthat onemay view them as proxies for a single underlying trait. It therefore suffices to sum theratings into a single scale or select any one rating and ignore the rest.18Perhaps the mostreasonable scale is the one derived from evangelism, distinctiveness, and pluralism, sinceHoge(1979: 192)describes these ratings as "[t]he actorsKelley stressed most."Hoge did not follow this approach,but instead ran separate zero-ordercorrelationsbetween denominational growth and all the variables. Working with the nine contextualvariables two at a time, he foundthat average family size and regional distributiontogetherexplained 59%of the variance in growthamongthe 16 denominations.Despite the fact thatmost strictness measures predictedmuch better than this, Hoge (1979: 190) took this resultas proofthat "contextualfactors alone can explain overhalf the total variance in denomina-tional growthrates in 1965-77."19His conclusionbears repeating:

... the main causation should be seen as beingfromcontextualfactorsfirst, then from nstitutional factors,not the opposite.The contextual factorstaken alone can explain over half the variancein denominationalgrowth n 1965-77.... This conclusion s somewhatdifferent romKelley'sview whichputs mostemphasison institutionalfactors.Perhaps Kelleyunderemphasized ontextual actors due to the hortatorypurposeofhis book ... Ourpurposeis explanation,not advocacy,and we estimate that contextualfactorscompriseover half the explanationfor denominationalgrowthor declinerates. (Hoge 1979:194-95)

210


16/21


But we have already seen that in cases like this a stepwise hierarchical approach s biasedeven if contextual factors are "causally prior"to institutional factors. Multiple regressionprovides a more appropriatebasis for inference, and at the very least one must check howsensitive the conclusions are to the orderin whichvariables enter the regressions.Table 5 shows the result of a more complete multiple regression analysis. The firstcolumn replicates Hoge's finding that the best two-variable combination of contextualvariables does indeed yield an R2 of .59.20But columns 2, 3, and 4 show that the strictnessratings do far better. The best rating, distinctiveness, explains 94% of the variance ingrowth.A comprehensiveadditive scale of all the ratings (except polity and ethnic identity)explains 83%.A three-item scale, based on the factors Kelley stressed most, explains 88%ofthe variance. Indeed,everyinstitutional variable (except ethnic identity and polity) explainsat least 70% of the variance in growth, more than the best two contextual variablescombined.21

TABLE 5DENOMINATIONALGROWTHREGRESSIONS

(1) (2) (3) (4) (5) (6) (7)CHILDREN .48* -.00 .11 .07

(2.73) (0.03) (0.85) (0.60)WEST .50* -.05 .22 .14(2.81) (0.47) (1.82) (1.26)DISTINCTIVENESS .97*** 1.01***

(15.10) (7.89)STRICTNESS1 .91*** .72***(8.65) (4.86)STRICTNESS2 .94*** .81***

(10.70) (5.71)adj-R2 .59 .94 .83 .88 .93 .85 .88Observations 16 16 16 16 16 16 16Note:Asterisks *, **,and ***denotesignificanceat the .05, .01, and .001 level, respectively.Beta coefficients n plaintext t-statistics in parentheses.Growth s the dependentvariable.Variable definitions:growth= percentagechangeindenominationalmembership,1965-75; West = percentageof the denomination'smembership iving in Westernstates;children= averagenumber of children in members' amilies; distinctiveness = expert scale; strictnessl = denomina-tion's mean scorefor distinctiveness, theology, evangelism, pluralism, social activism, and ecumenism;strictness2 =mean score fordistinctiveness,evangelism,andpluralism.Data Source:Hoge(1979).

The most important results are found in columns 5 through 7. These show thatcontextualeffectswash out of the equationonce the strictness variables enter. In every case,the contextual variables' betas and t-statistics approach zero, while the statistics for thestrictness variables remain extremelystrong.Figure 3 explains all. The relationship between growth and strictness is nearlyperfect. It is no wonder, therefore,that the contextual variables dropout - a single strict-ness scale derivedfromKelley'stheory, leaves nothing else to be explained!22In short, given the inherent limitations of the data, a more powerful empiricalvindi-cation of Kelley'stheory simply is not possible. Strictness measures trounce the competitionin this head-to-headtest. No otherconclusion s possible.23

211


17/21

JOURNALFORTHE SCIENTIFIC TUDYOF RELIGION

FIGURE 3GROWTHVERSUS STRICTNESS IN ACTUALDATA

40 -

Naz

AmBaptPresUSAmLuthLuthAmReformMeth Disc

UPres 33Strictness2

Data Source:Hoge(1979)CONCLUSIONS

Three major conclusions emerge from the preceding methodological review of theempirical research on church growth.First, like all social scientists, church growth researchers must pay careful attentionto the limitations of their data and methods. As computers have become vastly more power-ful and statistical programs more convenient, the dangers of data mining have increased.These problems are in no way mitigated by ever larger data sets and ever more sophisticatedcomputational techniques - indeed, larger data sets are more prone to throw up spuriousfindings, and sophisticated statistics (including most nonlinear procedures) tend to be moresensitive to specification errors than their simpler counterparts.

Second, statistical problems have almost certainly biased the results of past studies.Researchers have used data and methods that systematically privilege contextual factorsrelative to institutional factors. From an applied perspective, this means that church growthresearchers have understated the extent to which churches and their leaders can alter theirfate. From a theoretical perspective, it means that researchers have been too quick todismiss particular theories of religious organization (particularly those that trace thevitality of religious organizations to their strictness, costliness, sectarianism, and so forth).

Third, future research must not repeat the mistakes of the past, and whereverpossible old data must be reanalyzed using methods less prone to error and bias. This paper

AssemAdvent

LDSt'l

CO0

C)'.44

PI

20 -

O -

SoBapt

LuthMo

UCCEpis

-20 - I1 5 7I

212


18/21


undertook one such reanalysis, thereby reversing the original conclusion. The data frommost other studies still exist and should be subjectedto similar reanalyses. It is difficult topredict which findings will stand and which will fall. Some results will certainly proverobust,but many others may not. Despite a raft of reasonable-soundingconclusions,it is notyet clear that large data sets and complexstatistics have led to more precise and objectiveinsights than those derived from the personal experiences of church leaders and churchgrowthconsultants.Not all the news is bad. Each of the previously described pitfalls carries with it asolution or an alternative, many of them straightforward. I have already demonstrated asolution to the bias inherent in hierarchical models: Do not make stepwise comparisonsofR2to assess the relative impact of differentvariables. Wheremultiple factors are believed to beat work, limit the analysis to multiple regressions and accept the fact that, absent betterdata or a controlled experiment, the individual impact of some factors cannot be fullyassessed. The solution to errors in the independent variables and restrictions on theindependent variables is, whereverpossible, better data. Rather than ransacking a 500-itemsurvey distributedto thousands of members in a single denomination,direct a few carefullyworkedquestions (based on theory and past research) to a reasonable number of congrega-tions across a wide range of denominations. If at all possible, assess the level of responseerror in key items and take account of this error when drawing conclusions about therelative importance of different items. Where better data are unobtainable, identify thelimits and potential biases inherent in the current sample. Faced with a large number ofsurvey items, the researcher must resist the temptation to search for "significantpredictors."The most defensible approachis to identify a small number of test items andcontrols (implied by a carefullydevised, tightly stated theory)beforedoing any analysis. Allother items should be ignoredor introducedonly to check that the preselected variables aretruly robust. If this much self-restraint provesuntenable, then try the followingalternative:use a random sample of the data for ad hoc exploration and then test the apparentpredictors against the remaining data. Most of the spurious correlations will disappear inthe second stage and most of the true correlationswill remain.24If we do all these things, we may finally learn why churchesreally grow.We may evendiscover that both the reasons and the evidence were in plain sight all along.

NOTESThanks to RogerFinke, Daniel Olson,DarrenSherkat, RodneyStark, and three JSSR referees for commentsand suggestions.1 See also WhyConservativeChurchesReallyAre Growing:KelleyRevisited(Bibby1978)and 'Strictness'andChurchMembership, McFaul1974:281),which went so far as to claim that "[contraryto Dean Kelley'sthesis ... areturn to 'strictness' s the cause of, rather than the solution for, the mainline Protestantdenomination'membershipdecline."Forhis own part, Kelley(1978: 171)remained"unrepentantand unreconstructed."nskeep (1993) offersanexcellentsummaryandcritiqueof Kelley'scritics.2 See Roozenand Hadaway's 1993)bibliography or dozens of other books and articles on churchgrowth.3 Churchhistorian Martin Marty, admonished readers of the first volume to "notopen your mouth abouttrends and patterns in church membershipand participation unless you have read this book."Renowned churchgrowthconsultant Lyle Shaller declared of the second volume: "Never beforehas so much high quality research onchurch growth been available in one book.' And Ken Bedell, editor of the Yearbookof American and CanadianChurchescalled the second volume the "bestsociologicalanalysis to date on church and denominationalgrowth."4 Several studies addressedstrictness en passant. In the course of correlatingnumeroussurveyitems againstgrowth,they note which,if any, reflect some aspect of strictness. According o Kelley (1979: 340), however,the itemstested in this fashion bear little relationship o what he called strictness.

213


19/21

JOURNALFORTHE SCIENTIFIC TUDYOF RELIGION5 In a special issue of the Review of Religious Research that includes five essays about methodologicalproblems n congregationalstudies, Roozenand Carroll 1989: 116)note the continuing"lackof a cumulative,criticalmethodologicaliterature on the studyof congregations." heessays in the specialissue examine a varietyof samplingproblems(such as insufficient response rates, nonrepresentative samples, and difficulties identifying the relevantpopulation),but they do not addressesany of the statistical pitfalls that I will examine.Onereviewerhas stated thatthese pitfalls were "fully debated" and well known to "everyone"nvolved in the 1970s church growth research.Unfortunately, he debate never found ts wayinto print.6 Readersmay write the authorfor a disk andprintedcopiesof all the data and simulationprogramsdescribedin this paper. Thoughwritten forStata, they can be translated to SPSS, Systat, or any other full-featuredmicrocom-puterstatistics package.7 The sample restrictiondoes not bias the regressioncoefficient,but this is of little help since virtually all theinstitutional variablesin previousstudies are attitude scales that defyratio level measurement.In practice,therefore,churchgrowth researchers never concernthemselves with the numerical value of these coefficients,only their betavalues, and as Table 1 shows these betas arebiased downward.For a mathematical reatment of the rangerestrictionproblem,which verifies andextendsthe simulationresults, readersmay write forthe technicalappendix o this paper.8 Althoughthe specificnumbersobtainedabovedependon the assumptionsbuilt into the simulation,the basicresults remainvalid forvirtuallyany conceivable eal-world ituation. The technicalappendix,availableuponrequest,analyzes the mathematicsof rangerestrictions,and therebyverifiesand generalizesthe simulations.9 The problempersists when comparing he results of different,single-denomination tudies, even if some ofthe denominations n the studies are quite liberaland some quite conservative.As long as the congregationsof eachindividual denominationoccupya relativelynarrowsegment of the strictness spectrum, he observed strictness effectwithin each denominationwill be small. This fact may explain Roozenand Hadaway's 1993:42) claim that strictness"is unrelatedto growthwithin liberalor conservative amilies."10 The simulations generateGROWTH s a functionof 10 true causal variables(CAUSE1, . . , CAUSE10)and a large random error term. In addition to these causal variables, I generate290 "junk" ariables(JUNK 1, ...,JUNK 290) completelyindependentof GROWTH.CorrelatingGROWTHwith all 300 variables and retaining thosethat are "significant" t the .10 level typically yields a list of 20 to 30 variables,mostof which arejunk. The calculatedR2 exceeds the variables' rue predictivepower,manyjunk variablesappearto be moresignificantthan several of thetruecauses, and some causes nevermake the list.11 The typical study combinescensus data on communitydemographicswith churchrecorddata on member-ship and giving and survey responses (regarding he activities and attitudes of the members)froma small sample ofmembers and/or leaders. Sometimes a surveyis given to all membersattendingchurchon a given Sunday,but eventhis approachgreatlyoversamplesmore committedmembers.12 The algebraicderivations n the technicalappendixconfirmand extendthe simulationresults.13 Both sets of t-statistics are, of course,extremelyhigh because of the large numberof observationsand theperfect inearity of the simulation.14 For the sake of argument,I have not challengedthe notion that contextualvariablesare always "causallyprior" o institutional variables.Kelley(1979:338), however,has correctlynotedthat this is tantamount to assumingthat religion is "adependentvariable," ncapableof affectingthe environment n whichit thrives. Certainlyone canpoint to some instances where the order of causation runs fiom religion to demographics, as when a religionencourages arge families, healthy life-style,habits of thrift,or a commitment o high levels of education.15 The last two cases probablycome closest to capturingthe relationshipsenvisioned by most researchers.Althoughthe strength of the causal links are a matterof dispute, everyoneacknowledgeshe observed negative)corre-lation between income and strictness/sectarianism.Moreover, ormal theories of strictness/sectarianism derive thiscorrelationas a matter of logicalnecessity(Iannaccone1992,Stark andBainbridge1987).16 See Kennedy(1985: 69) fora briefdiscussionof omitted bias. See my technicalappendix or a mathematicalanalysis that generalizesthe simulationand confirms ts results.17 Hoge printedsome of the data in his chapterand listed the rest in a technicalappendix(Hogeand Roozen1979b:E1-E14). See Hogeand Roozen 1979a: 19) fordetails about its availability.Readerscan write me for the datain printed,spreadsheet,and Stata formats.18 The fact that six out of eight institutional attributes collapseto but one dimension s itself strong supportfor Kelley's thesis and its theoretical cousin, church-sect theory as advanced by Johnson (1963) and Stark andBainbridge 1985), and Iannaccone 1992, 1994).All these authorsapproachreligiousorganizationsfrom a unidimen-sional perspective.They order denominationsalong a single continuumof "strictness, ension, costliness"and explainthe social attributes of the religionin terms of its locationalong the continuum.A priori, there is no obviousreason

214


20/21


why the institutional attributes listed by Kelley and measuredby Hoge should all hang together.That they do hangtogether s evidence for the logicalcoherenceof Kelley'sthesis and modernchurch-sect heory.19 Hoge states most of his results in terms of simple and partial correlations,but in order to maintain

consistencywith the rest of this paper,I have translated his correlations nto their adjusted-R2 quivalents.20 Recall,however, hat this R2 is the productof a search overnine variables. Inrepeatedsimulations regress-ing the actual growthvariable onto nine randomlygeneratedvariables,the best two-variablemodel had an adjustedR over .22 and about one fifth of the regressionsyielding adjustedR2s over .40.21 Hoge(1979:192)recognized his and observed hat "the denominational haracteristicsattributedby Kelleyto growingdenominationsare strongly upheld by independent ratings of experts.The factorsKelley stressed most -emphasis on evangelism, emphasis on distinctive life-style and morality, and disallowingindividualism in belief-came out strongest."But since the context-onlyregressionshad already explained59%of the variance in growth,heinferredthat strictnesscould not possiblycount for morethan the remaining41%.22 Kelley(1979: 338) himself has pointedthis out, but the significanceof his critiqueseems to have been over-looked. He notes that the "threecontextualfactorsthe researchers oundstronglyrelatedto denominationalgrowth...

are said to 'explainover half the total variance n denominationalgrowthrates'- if theyare takenfirst. If the institu-tional factors ... are taken first, they explainvirtually all the total variance,and little is left forcontextualfactors toexplain!"23 The regressionsin Table 5 extend Hoge's analysis in a straightforwardmanner; hey do not representtheapproachto the data that I would recommend on theoretical and empirical grounds. Given the small number ofavailable observations and the fact that the study was designedto test Kelley'stheory,I would suggest approachingthe data as follows:First, attempt to extract fromthe institutional data a single variableor scale that best capturesKelley's notion of "strictness."Fortunately, since all but two of the institutional measures scale along a singledimension,there is no problemderivingsuch a scale (by simple summationorchoiceof a single item). Second,regressgrowthonto this "strictness" cale to see whether it does, in fact, predict growth.(It does, yielding R2s around .90.)Finally, introducethe contextual variables- one or two at a time - to see if the initial result, predictedby Kelley'stheory remains significant. (As it turns out, strictness not only remains "significant";t totally overwhelmsall othereffects, even when those othereffectsare introduced hrougha somewhatsuspect"search" rocedure.)24 This method cannot be applied retroactively to demonstrate the validity of an effect identified in priorcorrelations nvolvingall the data.

REFERENCESBangs, C. 1972. Deceptivestatistics. The ChristianCentury89 (August30):852-53.Bibby,R. D. 1978. Whyconservativechurchesreally are growing:Kelleyrevisited.Journal for the Scientific Study ofReligion 17(2):129-37.Finke, R. and R. Stark. 1992. The churching of America, 1776-1990: Winners and losers in America's religious

economy.New Brunswick,NJ: Rutgers.Hadaway,C. K. 1980. Conservatismand socialstrengthin a liberaldenomination.Reviewof ReligiousResearch21(3):302-14.___. 1989. Will the real SouthernBaptists please stand up:Methodological roblems n surveyingSouthernBaptistcongregationsand members.ReviewofReligiousResearch31(2):149-61.Hoge, D. R. and D. A. Roozen,eds. 1979. Understanding church growth and decline: 1950-1978. New York:ThePilgrimPress.Hoge, D. R. 1979. A test of denominationalgrowth and decline. In Understandingchurchgrowth and decline: 1950-1978,editedby D. R. Hogeand D. A. Roozen,179-97. New York:The PilgrimPress.__. 1979b. Technical appendix to understandingchurchgrowth and decline: 1950-1978. Hartford,CT: HartfordSeminaryFoundation.Iannaccone,L. R. 1992. Sacrifice and stigma:Reducingfree-riding n cults, communes,and other collectives. Journalof Political Economy100(2):271-91._ . 1994.Whystrict churchesare strong.AmericanJournal of Sociology99(5):1180-1211.Inskeep, K. W. 1993. A short history of churchgrowthresearch. In Church and denominationalgrowth, edited by D.Roozen and C. K. Hadaway,135-48. Nashville:AbingdonPress.Johnson, B. 1963. Onchurch and sect.AmericanSociologicalReview28:539-49.__. 1971.Churchand sect revisited.Journalfor the Scientific Study of Religion10:124-37.Kelley, D. M. 1972] 1986. Whyconservativechurches are growing:A study in the sociology of religion with a newprefacefir the ROSE Edition. Macon:MercerUniversity Press.___. 1978.Whyconservativechurches are still growing.Journalfor theScientificStudy ofReligion 17(2):165-72.__ . 1979. Is religiona dependentvariable? n Understandingchurchgrowthand decline:1950-1978,editedby D. R.Hogeand D. A. Roozen,334-43. New York:The PilgrimPress.

215


21/21

216 JOURNALFORTHE SCIENTIFIC TUDYOF RELIGION

Kennedy,P. 1985.A guide to econometrics.Cambridge,MA:The MITPress.McKinney, W. J. Jr. 1979. Performance of United Church of Christ congregations in Massachusetts and inPennsylvania.In Understandingchurchgrowthand decline:1950-1978, edited by D. R. Hogeand D. A. Roozen,224-47. New York:The PilgrimPress.McKinney,W. J. Jr. and D. R. Hoge. 1983. Communityand congregationalfactors in the growth and decline ofProtestant churches.Journalfor the ScientificStudyof Religion22: 51-66.McFaul,T. R. 1974. "Strictness" nd churchmembership.The ChristianCentury91 (March13, 1974):281-84.Olson, D. V. A. 1993. Congregationalgrowth and decline in Indiana amongfive mainline denominations.In Churchand denominationalgrowth,editedby D. A. RoozenandC. K. Hadaway,208-24. Nashville:AbingdonPress.Perry, E. L., and D. R. Hoge. 1981. Faith priorities of pastor and laity as a factor in the growth or decline ofPresbyteriancongregations.Reviewof ReligiousResearch 22: 21-32.Roof, W. C., D. R. Hoge, J. E. Dyble, and C. K. Hadaway. 1979. Factors producinggrowth or decline in UnitedPresbyteriancongregations.In Understandingchurchgrowthand decline:1950-1978,edited by D. R. HogeandD. A. Roozen,198-223. New York:The PilgrimPress.Roozen,D. A. and J. W. Carroll. 1989. Methodologicalssues in denominationalsurveys of congregations.ReviewofReligiousResearch31(2):115-31.Roozen,D. A., and C. K. Hadaway,eds. 1993.Churchand denominational rowth.Nashville:AbingdonPress.Stark, R.and W. S. Bainbridge.1985. Thefutureof religion. Berkeley:Universityof CaliforniaPress.. 1987. A theoryof religion.New York:PeterLang.Stamp,J. 1929. Some economic actorsin modern ife. London:Kingand Son.Thompson,W. L., J. W. Carroll,and D. R. Hoge. 1993.Growthor decline in Presbyteriancongregations.In Churchand denominationalgrowth,editedby D. A. Roozenand C. K. Hadaway,188-207. Nashville:AbingdonPress.Welch, M. R. 1993. Participation and commitment among American Catholic parishioners. In Chltrch anddenominational rowth,edited by D. A. Roozenand C. K.Hadaway,324-45. Nashville:AbingdonPress.

iannaccone - reassessing church growth dani

Documents