toward disaster-resilient cities: characterizing resilience of infrastructure systems with expert...

19
Risk Analysis, Vol. 34, No. 3, 2014 DOI: 10.1111/risa.12133 Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments Stephanie E. Chang, Timothy McDaniels, Jana Fox, Rajan Dhariwal, and Holly Longstaff Resilient infrastructure systems are essential for cities to withstand and rapidly recover from natural and human-induced disasters, yet electric power, transportation, and other infras- tructures are highly vulnerable and interdependent. New approaches for characterizing the resilience of sets of infrastructure systems are urgently needed, at community and regional scales. This article develops a practical approach for analysts to characterize a community’s infrastructure vulnerability and resilience in disasters. It addresses key challenges of incom- plete incentives, partial information, and few opportunities for learning. The approach is demonstrated for Metro Vancouver, Canada, in the context of earthquake and flood risk. The methodological approach is practical and focuses on potential disruptions to infrastruc- ture services. In spirit, it resembles probability elicitation with multiple experts; however, it elicits disruption and recovery over time, rather than uncertainties regarding system func- tion at a given point in time. It develops information on regional infrastructure risk and en- gages infrastructure organizations in the process. Information sharing, iteration, and learning among the participants provide the basis for more informed estimates of infrastructure sys- tem robustness and recovery that incorporate the potential for interdependent failures after an extreme event. Results demonstrate the vital importance of cross-sectoral communica- tion to develop shared understanding of regional infrastructure disruption in disasters. For Vancouver, specific results indicate that in a hypothetical M7.3 earthquake, virtually all in- frastructures would suffer severe disruption of service in the immediate aftermath, with many experiencing moderate disruption two weeks afterward. Electric power, land transportation, and telecommunications are identified as core infrastructure sectors. KEY WORDS: Disasters; expert judgment; infrastructure; interdependencies; resilience 1. INTRODUCTION Researchers and policymakers have called for concerted efforts to make cities and intercon- nected urban regions more “disaster-resilient.” (1–5) Although definitions of resilience differ, they im- ply that resilient cities can absorb shocks (from ex- School of Community and Regional Planning, University of British Columbia, 242-1933 West Mall, Vancouver, BC, V6T 1Z2, Canada. Address correspondence to Stephanie E. Chang, School of Com- munity and Regional Planning, University of British Columbia, 242-1933 West Mall, Vancouver, BC V6T 1Z2, Canada; [email protected]. treme events, such as natural disasters) while still maintaining function (in terms of providing the ba- sis for well-being of residents). Much of the current literature on urban disasters has emphasized land- use planning specifically, and hazard mitigation more broadly, for reducing disaster risk. (2,6–9) An emerg- ing body of research addresses planning for postdis- aster reconstruction and recovery, emphasizing the rebuilding of communities’ social as well as phys- ical fabrics, (5,10,11) yet the resilience of cities after a disaster is largely determined by the functioning of complex, interdependent infrastructure systems. Godschalk (3) suggests that for cities to be resilient, their roads, utilities, and other infrastructure systems 416 0272-4332/14/0100-0416$22.00/1 C 2013 Society for Risk Analysis

Upload: holly

Post on 19-Feb-2017

222 views

Category:

Documents


8 download

TRANSCRIPT

Page 1: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Risk Analysis, Vol. 34, No. 3, 2014 DOI: 10.1111/risa.12133

Toward Disaster-Resilient Cities: Characterizing Resilienceof Infrastructure Systems with Expert Judgments

Stephanie E. Chang, Timothy McDaniels, Jana Fox, Rajan Dhariwal, and Holly Longstaff

Resilient infrastructure systems are essential for cities to withstand and rapidly recover fromnatural and human-induced disasters, yet electric power, transportation, and other infras-tructures are highly vulnerable and interdependent. New approaches for characterizing theresilience of sets of infrastructure systems are urgently needed, at community and regionalscales. This article develops a practical approach for analysts to characterize a community’sinfrastructure vulnerability and resilience in disasters. It addresses key challenges of incom-plete incentives, partial information, and few opportunities for learning. The approach isdemonstrated for Metro Vancouver, Canada, in the context of earthquake and flood risk.The methodological approach is practical and focuses on potential disruptions to infrastruc-ture services. In spirit, it resembles probability elicitation with multiple experts; however, itelicits disruption and recovery over time, rather than uncertainties regarding system func-tion at a given point in time. It develops information on regional infrastructure risk and en-gages infrastructure organizations in the process. Information sharing, iteration, and learningamong the participants provide the basis for more informed estimates of infrastructure sys-tem robustness and recovery that incorporate the potential for interdependent failures afteran extreme event. Results demonstrate the vital importance of cross-sectoral communica-tion to develop shared understanding of regional infrastructure disruption in disasters. ForVancouver, specific results indicate that in a hypothetical M7.3 earthquake, virtually all in-frastructures would suffer severe disruption of service in the immediate aftermath, with manyexperiencing moderate disruption two weeks afterward. Electric power, land transportation,and telecommunications are identified as core infrastructure sectors.

KEY WORDS: Disasters; expert judgment; infrastructure; interdependencies; resilience

1. INTRODUCTION

Researchers and policymakers have called forconcerted efforts to make cities and intercon-nected urban regions more “disaster-resilient.”(1–5)

Although definitions of resilience differ, they im-ply that resilient cities can absorb shocks (from ex-

School of Community and Regional Planning, University ofBritish Columbia, 242-1933 West Mall, Vancouver, BC, V6T 1Z2,Canada.∗Address correspondence to Stephanie E. Chang, School of Com-munity and Regional Planning, University of British Columbia,242-1933 West Mall, Vancouver, BC V6T 1Z2, Canada;[email protected].

treme events, such as natural disasters) while stillmaintaining function (in terms of providing the ba-sis for well-being of residents). Much of the currentliterature on urban disasters has emphasized land-use planning specifically, and hazard mitigation morebroadly, for reducing disaster risk.(2,6–9) An emerg-ing body of research addresses planning for postdis-aster reconstruction and recovery, emphasizing therebuilding of communities’ social as well as phys-ical fabrics,(5,10,11) yet the resilience of cities aftera disaster is largely determined by the functioningof complex, interdependent infrastructure systems.Godschalk(3) suggests that for cities to be resilient,their roads, utilities, and other infrastructure systems

416 0272-4332/14/0100-0416$22.00/1 C© 2013 Society for Risk Analysis

Page 2: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 417

must be designed to continue functioning under ex-treme hazard conditions; further, public and privatesector organizations must be prepared, have up-to-date information, be connected by effective commu-nication networks, and have experience in workingtogether.

Infrastructure systems are highly vulnerable indisasters and their failures lead to widely felt losses.In the 1994 Northridge (Los Angeles) earthquake,for example, Gordon et al.(12) estimate that nearlyone-quarter of business interruption losses could beattributed to failure of highway bridges. Infrastruc-ture systems are also highly interdependent. Nojimaand Kameda(13) document the range of infrastructuredisruptions caused by electric power outages afterthe 1995 Kobe (Japan) earthquake. Similar interde-pendencies and associated societal disruptions havebeen observed in hurricanes, ice storms, blackouts,terrorism events, and other types of disasters.(14–16)

In broad terms, the tasks facing risk managersin efforts to foster urban resilience regarding infras-tructure systems have at least two components: (i)characterizing vulnerabilities and resilience1 withinexisting systems to disasters, and (ii) setting prior-ities for mitigation efforts to improve resilience.2

Practical methods that can be applied in cities toaddress these issues are urgently needed. Yet thesetwo tasks entail some unusual complexities: theymust address multiple sources of disasters, multiplepathways for system failure, multiple and cascadinginterdependencies among a wide array of infras-tructure systems, and many potential alternativemeasures to reduce failure risk within and acrosssystems, in the context of ill-defined and multilay-ered governance structures. All these complexitiesmust be addressed within existing governance andplanning processes, in which many of the infrastruc-ture systems are privately owned or operated, andsecurity concerns make information sharing difficult.These tasks may not be familiar or easy ones forrisk managers, but in our view hold the potential toovercome structural obstacles and help make infras-

1While vulnerability refers to the propensity for loss in the eventof a hazard, resilience refers to the capability of withstanding andrecovering quickly from such an event. These concepts could beconsidered at levels ranging from individuals to nations. Herewe address urban resilience with a focus on sets of infrastructuresystems.

2Other challenging tasks include (i) obtaining funding for long-term risk reduction, which competes with near-term operatingand service priorities, and (ii) implementation in upgrading orsiting new facilities. We do not address these topics here.

tructure systems, and thus cities, more resilient todisasters.

The objective of this article is to develop anddemonstrate a practical approach for risk managersworking with infrastructure systems and communi-ties to characterize vulnerability and resilience of in-frastructure systems in disasters. We stress that this isan exploratory effort, based on two cases conductedsequentially. Our intent is not to provide general-ized advice on what particular steps will make re-gions more resilient to infrastructure failures, sincethese prescriptions will be particular to and het-erogeneous across regions. Rather, the intent is toexplore and apply nonprobabilistic, judgment-basedapproaches for characterizing resilience of systems.This approach is informed by the writing on prob-ability elicitation(17–19) but applied to dynamic pro-cesses. We also rely on elements of collaborative de-cision processes that provide a context for informa-tion sharing and learning(20,21) to help overcome in-trinsic barriers that inhibit collective action to man-age the interdependencies of infrastructure failuresin disasters.

The approach is nonprobabilistic. Risk is some-times defined as a triplet of conditions: what couldgo wrong, how likely it is to go wrong, and the con-sequences if it does go wrong;(22) here, we addresstwo of these three questions. We rely on disasterscenarios to address what could go wrong in termsof an extreme event. We develop approaches thatmake use of historical experience with similar haz-ards in other contents, and the judgments of technicalspecialists to characterize what critical infrastructureservices could be lost in a major disaster, to what ex-tent, and for how long. We also investigate throughinterviews and workshop processes how disruptionsin one infrastructure sector could cause ripple ef-fects on other downstream sectors. We use thesefindings to characterize consensus views on effectson regional services and their broad consequencesfor regional residents. We neither attempt to char-acterize the likelihood of the initiating event (e.g., anearthquake of a given magnitude and location) norare regional data available to characterize the prob-abilities of system failures, and then interdependentfailures in other systems, conditional on that event.Rather, we rely on a nonprobabilistic judgmentalcharacterization, informed by feedback and makinguse of the views of several specialists, to providepoint estimates of resilience of systems, as a basisfor subsequent planning and decision processes formitigation.

Page 3: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

418 Chang et al.

This article makes three contributions. First, weoutline a set of challenges that lead to an ongoing“market failure” in terms of information availabilityand thus investment to address resilience of infras-tructure systems at the community or regional level.These challenges motivate the need for communica-tion among infrastructure system owners about theresilience of their individual systems in a given event,which will necessarily be based on judgments, giventhe lack of shared knowledge about the resilience ofother systems. Second, we develop and apply an ap-proach to expert elicitation of resilience in terms oftwo variables, service disruption and recovery overtime. Although elicitation of probabilities from ex-perts has been practiced for decades,(19,23) our iter-ative nonprobabilistic approach, focusing on disrup-tion and recovery over time, is a new opportunityand approach for expert elicitation. Third, the worksets out a series of steps for iteratively characterizing,communicating, and updating the overall resilienceof the set of infrastructure systems in a region in re-sponse to a scenario of an extreme event.

This article is structured in the following man-ner. Section 2 discusses four concepts that are crucialto this effort: a brief review of existing treatmentof resilience and interdependency in sets of urbaninfrastructure systems; the obstacles to effectivemanagement of infrastructure interdependencies inextreme events; the importance of collaborative in-formation sharing to help overcome these obstacles;and the role of expert judgments in such assessments.Section 3 discusses the methodological approachwe developed to characterize infrastructure vulner-ability and resilience, emphasizing infrastructureinterdependencies. This approach has similaritiesto methods for eliciting probabilities (a well-knowntechnique in decision analysis and risk analysis), butwith some important differences. We apply this ap-proach in two different cases in Vancouver, BritishColumbia. Section 4 summarizes the results andsynthesizes over the two cases. Section 5 providesdiscussion and conclusions.

2. CONTEXT AND LITERATURE

2.1. Resilience Within Infrastructure Systems

Resilience of complex systems (the ability to ab-sorb shocks while maintaining function) has emergedas a fundamental concern for systems managers andresearchers(24) and for those affected by system fail-

Fig. 1. Resilience concept.

ure, such as elected officials, agencies, private or-ganizations, and civil society. Much of the inter-est in resilience has arisen in work regarding com-plex social-environmental systems, although con-cepts of resilience are also directly relevant for en-gineered systems,(25) including infrastructure systemswithin communities.(16,26,27)Because of the potentialfor large-scale infrastructure failure interactions andthe spatial implications of infrastructure system fail-ures, a community or regional perspective on in-frastructure resilience would be highly useful. Yetmuch of the work on resilience in regional contextseither is theoretical, with a focus on the role ofinstitutions,(28,29) modeling-oriented, often with a fo-cus on specific subsystems,(30,31) or aimed at under-standing resilience of economic entities and systemsto infrastructure disruption, rather than the resilienceof the infrastructure systems themselves.(31,32) Thereis a need for methods to characterize infrastructureresilience in applied, real-world contexts, based onthe limited information at hand regarding perfor-mance in extreme events.

Fig. 1, adapted from Bruneau et al.(26) andMcDaniels et al.,(16) provides a schematic represen-tation of the resilience concept in a particular in-frastructure system in terms of system function (e.g.,the number of customers served by a municipal wa-ter system) before and after an extreme event. Attime t0 shown on the graph, an extreme event suchas an earthquake occurs. The degree to which sys-tem function is not forced to zero at that point rep-resents the system robustness, or ability to withstandan extreme event of a given level and still maintainsome degree of system function. Over time, systemfunction recovers until at tR it is fully restored. (Note

Page 4: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 419

that in some cases, the new system configuration maydiffer from the preevent configuration, as discussedin Holling.(25)) Increasing resilience, which is equiva-lent in the diagram to reducing the loss triangle, canbe accomplished through ex ante measures such asinstalling backup generators for a hospital or ex postactions such as rerouting traffic.

Addressing infrastructure interdependencies iscentral to fostering infrastructure resilience. Yet theemerging literature on infrastructure interdependen-cies has been dominated by systems engineering ap-proaches in which the goal is optimization, ratherthan resilience. These engineering studies have fo-cused on the technical complexities of interdepen-dent systems. Some have developed computer-basedsimulation models of infrastructure systems and theirlinkages.(33,34) Others have proposed model repre-sentations and analytical approaches to character-ize interdependencies and identify key vulnerabili-ties, particularly from terrorism threats.(35,36) Link-ages between infrastructure and social systems arerarely considered; where they are, these are limitedto modeling how physical infrastructure loss wouldaffect economic sectors.(34,37) Decision making is gen-erally addressed indirectly, if at all, through mathe-matical optimization; for example, by identifying sys-tem links that would have the greatest consequencesif damaged in an extreme event.

To foster infrastructure resilience, research isneeded that links urban physical systems with hu-man communities, that supports the informationand communication needs of infrastructure organiza-tions, and that directly addresses infrastructure deci-sion making at the urban and regional scales.3 A firststep in such research is to develop methods to charac-terize how a given region may be vulnerable to the di-rect loss of infrastructure system services in a specificextreme event, and also vulnerable to infrastructurefailure interdependencies (IFIs) that prolong and ex-tend the loss of services. (IFIs occur when failure inone system leads to failures in other, dependent sys-tems; e.g., when power outages lead to malfunctionof pumps in water systems.(16) These methods shouldalso characterize how much infrastructure systemscan recover function to the extent possible within agiven period of time (as in Fig. 1). In addition to this

3Such studies on infrastructure interdependencies are rare. Oneexample is work by McNally et al.(38) that develops a GIS-basedinformation system to help infrastructure professionals learnabout the behaviors of interdependent infrastructures. The ap-proach incorporates visual representations, disruption scenarios,and expert knowledge.

resilience information, a second need is to foster thecapacity of infrastructure organizations to addressrisk. Berke and Campanella(5) suggest that foster-ing community resilience in general requires buildingnetworks among community groups, in part to relayinformation related to risk and risk reduction. Theyobserve that where groups have no history in work-ing together, intermediary groups may be useful tofoster attention and agreement.

2.2. Obstacles to Fostering Resilience

The obstacles to fostering infrastructure re-silience are not trivial, particularly when interde-pendencies are acknowledged. Three challengesare particularly noteworthy. First, organizationalinterests are often at odds with regional interests.The objectives of infrastructure managers, insofar asdisaster risk is concerned, include reducing physicaldamage to their own infrastructure system, minimiz-ing investment and repair costs, minimizing revenuelosses, and maintaining the organization’s reputa-tion. Private infrastructure providers are furtheraccountable to their shareholders. Infrastructureproviders have few incentives to be concerned withthe effects of own-system disruptions on other,dependent infrastructures (referred to here asdownstream effects). In the case of electric powerblackouts, for example, the current legal standardsand precedent are such that it is not the electricpower utilities but rather the dependent serviceproviders (e.g., building maintenance, in the case oflighting for emergency stairwells) that are liable forconsequent losses (e.g., injury from stairwell fallsin power outages).(39) Thus infrastructure providershave little incentive to understand the downstreameffects of their system’s outages and to considerthem in decision making. Actions such as retrofittinginfrastructure that may have large societal benefits,especially through reducing infrastructure failureinteractions, may not be undertaken because theyare not sufficiently cost effective or beneficial to theinfrastructure organization itself.(40)

A second challenge pertains to security concernsand barriers to information sharing. Infrastructureorganizations are well aware that their systems rep-resent prime targets for acts of terrorism, and somay be reluctant to share information about systemvulnerabilities. After the attack of September11, 2001, public availability of information aboutcritical infrastructure has become much morerestricted.(41,42) This means that infrastructures that

Page 5: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

420 Chang et al.

are vulnerable to disruptions in other systems (herereferred to as upstream disruptions)—hospitals de-pendent upon electric power, for example—have dif-ficulty gathering information about what to expect infuture disasters as a basis for their preparedness plan-ning. (This is an even greater problem for communitygroups than for downstream infrastructures.)

A third challenge derives from the observationthat few infrastructure managers have direct expe-rience with major disasters. For example, even inLos Angeles—a city with high seismic hazard—thelast two major earthquakes took place in 1994 and1971. Our case study region, Metro Vancouver, isin a moderate-to-high seismicity zone, but no ma-jor earthquakes have occurred in the region in liv-ing memory. The concept of adaptation and learningfrom experience is central to concepts of resilience.(3)

The paucity of direct experience suggests the impor-tance of learning from other regions’ experiences ofdisaster.

To summarize, these three challenges (partialincentives, limited and asymmetric information, andlack of experience) create a form of market failure,in which it is difficult or impossible for individualinfrastructure operators to understand their poten-tial robustness to an extreme event and resiliencewithin a set of regional infrastructure systems. As aresult, there will be underinvestment in efforts to im-prove the resilience of individual systems, and thusthe set of regional systems, compared to the level thatwould be socially optimal. An individual firm’s busi-ness continuity planning cannot be expected to over-come these challenges because of the lack of knowl-edge about the performance of other systems, andthe differences between private and social incentivesregarding mitigation investments.

The approach we develop in this article ad-dresses these three challenges through structureddata gathering, elicitation of judgments, informationsharing, and integrated analysis in a collaborative ap-proach, as discussed later.

2.3. Planning for Infrastructure Resilience

The planning and risk management litera-tures have long emphasized the conceptual meritsof collaborative approaches to decision making,often involving civil society groups, as well astechnical specialists and agency staff.(20) Althoughmany important benefits of collaborative planninginvolving citizens and technical specialists havebeen articulated,(43) the experience in practice has

been mixed, although still encouraging, based onevaluations of citizen involvement in complex envi-ronmental and technology issues. One notable aspectof collaborative planning is the appropriate role oftechnical specialists who provide and share technicalinformation regarding potential alternatives andtheir consequences to address a policy question.(44)

The role of technical specialists with understandingof specific infrastructure systems is crucial whenconsidering vulnerability and resilience of systems tonatural disasters. In sum, we believe that collabora-tive approaches are crucial for overcoming the threechallenges discussed to building resilience.

2.4. Reliance on Judgments

All risk analysis and risk management re-lies on judgments regarding both technical issuesand preference issues, made either implicitly orexplicitly.(45,46) One approach with wide recognitionand application in decision and risk analysis is elici-tation of probabilistic expert judgments, typically ex-pressed as a cumulative density function over a well-defined quantity that is specified in time and space,made in response to a set of conditioning assump-tions, within a predefined structure.(19,23,47)Manyexamples of probability elicitations have been pub-lished. The judgment tasks explored in this studyemploy the basic concepts and structure of proba-bility elicitation, but with an important difference,which can be understood with reference to Fig. 1.In contrast to Fig. 1, probability elicitation employsa cumulative probability density function (cdf) torecord and communicate expert judgments regardinguncertainty for a precisely defined variable, typicallyat one point in time (unless duration is the variable).If our focus were probability elicitation, then thestructure of Fig. 1 would be quite different andwould require multiple representations. A proba-bilistic version of Fig. 1 would show either the cdffor the system function at a point in time, or thecdf of duration to achieve a given level of recoveryof system function, which could be measured interms of percentage of service area or customeraccounts served. In our approach, Fig. 1 is dynamicover time, but shows no uncertainty regarding thelevel of service at each point in time. Of course, thepsychological nature of the judgment tasks is alsodifferent. In our view, the judgment tasks requiredto construct Fig. 1 are highly relevant for the issuesof regional infrastructure resilience, and for under-standing potential IFIs, because resilience (in terms

Page 6: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 421

of robustness and recovery time) is elicited directly.The kinds of information in Fig. 1 are particularlyrelevant given the purpose of the assessments is tobroadly inform understanding of regional infrastruc-ture resilience through considering the resilience ofthe individual infrastructure sectors.

3. METHODS

3.1. Overview

The sequential protocols for probability elicita-tion typically involve some version of the followingsteps:(19,47) structuring, conditioning, motivating, elic-iting, revising, and verifying.4 These steps all involvejudgments by the elicitor regarding appropriateprocesses to follow, in order to achieve more man-ageable and consistent judgment tasks for the expert,in hopes of improving the quality of elicitationresults. Because judgments about risk issues aresubject to considerable influence from heuristics andbiases,(48) the processes often make use of questionstructures to help overcome well-known biases.(19)

The psychology of behavioral judgments regardingissues of resilience within complex systems is far lesswell defined or explored in research than for theuncertainties typically addressed with expert elicita-tion. However, one could expect that overconfidenceby experts and a lack of alertness to a wide range offailure modes would be key concerns. The judgmenttasks for resilience of a region are made even morecomplex because of heterogeneity in both the initialeffects of an extreme event over space, and the rapid-ity with which recovery occurs over space and time.

Although these complexities are daunting, thepurposes and uses of the analysis should help keepthe complexities in perspective. We are not attempt-ing to model the spatial patterns of specific systeminterdependencies and failure patterns in order to,say, estimate economic losses from a given extremeevent. Rather, we seek to explore broad patterns of

4The approach of Cooke(23) has an additional step at the be-ginning of the process, which focuses on calibration (gaugingjudgment quality) as a basis for selecting experts to serve asthe source of judgments. The approach is clearly relevant forrepeated judgment contexts with rapid feedback (e.g., weatherforecasting). In considering judgments for rare events, the ap-proach relies on answers to almanac-like questions, to infer somegeneral level of calibration. For this study, we worked with thebest available experts, for each type of infrastructure system, se-lected by the infrastructure system operators. Hence, we had nodirect basis for or need for addressing generalized calibration.

resilience over whole regions, with a particular em-phasis on IFIs, in order to help inform ex ante under-standing of regional resilience and ultimately pointtoward opportunities for low-cost, high-consequencemitigation opportunities.

Our approach for eliciting resilience judgmentsadopts a similar structure to probability elicitationprotocols, with some important differences. We havefour linked sequential phases: (1) structuring andconditioning, in which we develop and test waysto structure the expert judgments, develop specificdetailed hazard scenarios, and conduct backgroundresearch about similar hazard events; (2) expertinterviews, in which we motivate the process withdiscussion of why this issue is important, and elicittheir preliminary judgments on resilience of onespecific infrastructure, operated by their organi-zation; we also interview experts about potentialinteractions with other infrastructure systems; (3)data synthesis, in which we assemble the resultsof several interviews into a set of diagrams; and(4) information sharing, feedback, and revisions, inwhich the infrastructure system operators and theanalysts meet together, to allow information sharing,and updating of the original estimates, based onlearning about the vulnerability and resilience ofother systems. These steps are summarized in Fig. 2and discussed in more detail later.

We have conducted this whole process twice,once in 2008 addressing an earthquake scenario, andonce in 2009 addressing a flood scenario, both inreference to the Metro Vancouver region of BritishColumbia. We refined the methods over the courseof the two processes. In what follows, we present asynthesis of the methods as they have evolved.

3.2. Structuring and Conditioning

3.2.1. Scenario

One initial step is to develop a hypothetical butrealistic hazard scenario that characterizes a signif-icant threat to the region. In our two studies of2008 and 2009, we employed one scenario in eachyear (one earthquake, one flood). Here we providea brief overview of the earthquake scenario used inthe 2008 work.5 This earthquake scenario was used

5In the ideal situation, it would be preferable to condition thejudgment tasks on a range of different earthquake scenarios, toexplore thresholds in consequences. The obvious tradeoff thatarises in exploring more scenarios is the time and effort re-quired from the experts to complete the elicitation tasks. The

Page 7: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

422 Chang et al.

Fig. 2. Methodological approach.

in both the expert interviews and the workshop pro-cess to provide a realistic conditioning event for thejudgment tasks and frame a consistent discussionfor each of the infrastructure representative inter-views. The specific scenario selected—an M7.3 shal-low, crustal earthquake with epicenter near Vancou-ver in the Strait of Georgia—was one that the BritishColumbia provincial government had been develop-ing for regional emergency planning purposes. Anearthquake of this location and magnitude is realisticfor the study area, although it does not represent aworst-case event. Working with the Provincial Emer-gency Program’s seismologist, we developed a mapof ground shaking intensity for the event and an as-sociated one-page description of the event includinglimited damage predictions. Modified Mercalli Inten-sity (MMI) levels for the scenario earthquake rangefrom VI to VIII in the majority of the study area (aregion of 2.2 million people), indicating strong to se-vere shaking levels in specific locations depending onsoil types.

3.2.2. Previous Experience

We conducted research regarding the infrastruc-ture failures and interdependencies that have oc-

individuals we interviewed are employees of infrastructureproviders with no obligation or incentive to participate in thiswork, other than the opportunity to learn from the process.

curred in previous earthquake disasters. This reviewcovered 10 critical infrastructure sectors as definedby Infrastructure Canada,6 an agency of the federalgovernment. The summaries focused on urban earth-quakes that had occurred in industrialized countrieswith similar development patterns and infrastructuresystems to those in Canada.7 In addition to infor-mation on past earthquakes, we explored a databaseregarding infrastructure interdependencies in non-earthquake disasters that we developed for relatedresearch.(16,49) The hazard scenario and summariesof other disasters were provided to the experts be-fore the interviews discussed later. These were in-tended to help expand their thinking about the rangeof infrastructure interactions, as well as inform themabout the extent of loss of system function, and timeto recovery, which have actually occurred in previ-ous events comparable to the scenario. These effortswere intended to help overcome the lack of expe-rience with earthquakes or other extreme events inthe Vancouver region, and bring available informa-tion from elsewhere to the local experts.

6These included: energy and utilities, communications and infor-mation, finance, healthcare, food, water and wastewater, trans-portation, safety, government, and manufacturing.

7They included the M6.9 1995 Kobe, Japan, earthquake; the M6.71994 Northridge (Los Angeles) earthquake; the M6.9 1989 LomaPrieta (San Francisco Bay Area) earthquake; the M6.6 2004Niigata Ken Chuetsu, Japan, earthquake; and the M6.8 2001Nisqually (Seattle) earthquake.

Page 8: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 423

3.3. Expert Interviews

Expert interviews are the primary means of datacollection in this approach. Recruitment of expertswas conducted on the basis of two principles: first,that the interviewees be the most knowledgeable per-sons in their organizations to provide the informationrequested; and second, that the experts collectivelyrepresent the entire cross-section of interdependentcritical infrastructure sectors. Sectors represented inthis study included: utilities (electric power, water,wastewater, natural gas); transportation (bridges andhighways, public transit, airports, seaports); telecom-munications; healthcare (regional health authori-ties, hospitals); and provincial, regional, and localgovernments.

Appropriate experts within each infrastructureorganization were identified on the basis of theirprofessional roles, as well as through recommenda-tions from other infrastructure experts. Individualsselected for the interviews were organizationalspecialists on emergency response or infrastructureengineering and system performance, or both. Often,they were the only individuals within the organiza-tions with the knowledge and experience to providethe expert judgments sought. Note that an expertinterview approach is not based on ideas of a repre-sentative sample within a sampling frame. Rather,each expert provided the best available informationabout his/her own infrastructure system, often basedon input from others in the organization. A totalof 13 interviews involving 18 professionals wereconducted in person, always with two individualsfrom our research team. With a single exception (amunicipality), all organizations contacted for thisstudy agreed to participate. Interviews typicallylasted one to two hours and interview notes weresent back to all participants for verification.

The main objective of the interviews was togather information on the resilience of the infras-tructure systems in terms of their ability to with-stand and rapidly recover from extreme events, andto characterize steps that can be taken to increase thisability. Our questions were intended to elicit judg-ments about how the particular system would func-tion, in terms of service provision, at three time pe-riods: just after an extreme event, three days later,and two weeks later (see also the interview script inthe Appendix).8 We also inquired into interdepen-

8We did not ask for information on the location of specific in-frastructure facilities. This helped to overcome the infrastructure

dencies among systems, in terms of other systems onwhich the given system depends for inputs and theextent to which other systems depend on the givensystem for outputs in order to function. We askedquestions about that system’s planning for extremeevents, about the interviewees’ sources of informa-tion and the degree to which they participate in mul-tisectoral discussions about performance of interde-pendent systems in extreme events. We were particu-larly interested in ways to reduce regional vulnerabil-ity to interdependencies among infrastructures. Allinterviews began with a discussion of the hypotheti-cal scenario and its associated map. In the 2009 inter-view process, we developed a table to more directlycapture and summarize the resilience judgments. Wealso developed tables to elicit how one sector (e.g.,electric power) is expected to perform by other sec-tors dependent on it (see interview script in the Ap-pendix).

We are aware of the potential for overconfidencein such judgment tasks. We cautioned the expertsabout this potential problem in keeping with othersuch protocols,(19) and also informed them of the ex-perience of infrastructure interdependencies in othersimilar events, noted earlier. We believe our iterativeprocess, with subsequent meetings and feedback re-garding the judgments of other experts, helped lessenthe potential for overconfidence, compared to a one-time interview. The interview format was pretestedwith an informed colleague who is also an expert onone of the region’s major infrastructure systems, andrevised somewhat before implementation.

3.4. Data Synthesis

Information from the interviews was synthesizedinto two types of diagrams that initially documentedthe interview results and later served as the ba-sis for cross-sectoral discussion and revision. Thefirst, service disruption diagrams, provide a regionaloverview of expected system function loss and recov-ery over time for all the infrastructure sectors inter-viewed. These diagrams summarize information onrobustness and recovery time that helps characterizesystem resilience as depicted conceptually in Fig. 1.In the initial version of these diagrams, informationfor each sector was provided by the informant forthat sector; the final version incorporated revisions atthe workshop. The second, interdependency diagrams

representatives’ potential concerns regarding security and confi-dentiality of information.

Page 9: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

424 Chang et al.

Fig. 3. Classification of service disruption levels.

(in the spirit of Rinaldi et al.(50)), captured the func-tional interdependencies among sectors and helpedin visualizing how disruptions in one sector might af-fect other dependent infrastructures. These two typesof diagrams are shown and discussed in Section 4(Results).

This synthesis required a succinct scale tobroadly characterize the societal impacts from thesystem function loss, which would be applicable forall sectors. We adapted a scale employed in Changet al.,(51) as well as input from the infrastructureprofessionals, to develop the scale shown in Fig. 3.Overall service disruption is rated according to fourlevels: “no loss,” “low disruption,” “moderate dis-ruption,” and “severe disruption.” Disruptions arerated by characterizing two dimensions—spatial ex-tent and severity of impact—along a scale that rangesfrom “low” to “high.” Impact considers the severityof consequences and the duration of the disruption.For example, complete loss of water supply indicatesa high impact (independent of how many peopleare affected), whereas a boil-water advisory lastinga couple of days represents an inconvenience, orlow impact. Extent considers the spatial reach of thedisruption as well as the proportion of people withinthe area that are affected. A disruption that causeda single death, for example, would be considered tohave high Impact but low Extent. Fig. 3 translates themeasures of extent and impact into an overall ratingof service disruption: infrastructure outages that

have both high extent and high impact, for example,are considered “severe” disruptions.9

When providing their judgments about levels oftheir system’s function over the whole region, at thetime of the event and up to two weeks after that,the participants were told to condition their assess-ments on the state of their systems as they exist atpresent, the stated disaster scenario, and assumptionsregarding the “rest of the world” outside of the af-fected areas as it exists at present. The experts weretold to average the performance of their system overthe whole region, recognizing some areas (e.g., thosewith more intense shaking) would be more severelyaffected, and some areas less affected. This condi-tioning of judgments on assumptions is crucial in ex-pert elicitation.

3.5. Information Sharing, Feedback, and Revision

The final component of the approach involveda one-day invitational workshop, held in Vancouver,British Columbia. The 2008 event regarding theearthquake scenario involved 13 infrastructure rep-resentatives, including most of those who had beeninterviewed, as well as others who had been unableto participate in the interviews. The participantscollectively represented a substantial proportionof major infrastructure organizations in the region.The purpose was to (1) provide all participants withan overview of the interview findings, (2) provideopportunities for feedback and revision, and (3) de-velop a consensus perspective on potential IFIs andsystem resilience based on the earthquake scenario.

The research team began with a review of the dis-aster scenario, and then summarized key interview

9Cox(52) has critiqued the use of categorical, qualitative risk ma-trices, with probability and consequence as two dimensions ofthe matrix, which at first glance might resemble Fig. 3. In fact,our approach differs from that of concern to Cox in several ways.It is conditioned on an earthquake or flood scenario. Given thisevent, we seek to characterize the consequences on society intwo dimensions—the areal extent and severity of impact—withno consideration of probability. Although some of the criticismsraised by Cox may also be applicable to our approach (such aslimited resolution), we do not believe the risk of technical errorsis any greater than with any other possible approach, given thecomplete lack of other sources of information about resilienceof systems after an extreme event. We recognize these broadcategories do not provide details of impacts, but rather are in-tended for overall comparisons, in broad classifications, acrossmany kinds of systems. Hence the potential uses of the infor-mation from this work offset to some degree the low precisionof the results. The method also recognizes tradeoffs between in-sight, precision, and effort, given the information at hand.

Page 10: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 425

findings using the service disruption and interdepen-dency diagrams. Through discussion and workbookexercises, participants were asked to consider, revise,and augment the information captured by these di-agrams. Section 4 provides a discussion of the revi-sions to the diagrams.

4. RESULTS

In what follows, we emphasize the results ofour 2008 project regarding earthquake, with someobservations from our 2009 work on floods addedwhere noted. Overall, the 2008 interviews indicatedthat IFIs receive widely varying levels of attentionin current preparedness and mitigation efforts.When asked about the organizations’ sources ofinformation about potential disruptions to upstreaminfrastructure sectors, 62% (8 of 13 interviews)mentioned industry associations, consulting studies,publications, and similar sources that may provideexperience-based information from other disasters.Some 54% indicated cross-sector discussions withspecific infrastructures in the region. Only 31% in-dicated drawing information from both experience-based sources and regional cross-sectoral discussion.When asked about specific cross-sectoral discussionsin their planning processes, interviewees mentionedon average just two other sectors, most commonlyfederal, provincial, and/or local governments (54%)and electric power (50%). These discussions arelargely bilateral. Only 46% mentioned regionalemergency preparedness and/or regional infrastruc-ture coordinating bodies. Moreover, intervieweeswere able to identify “upstream” infrastructure sec-tors (those on which they depend) with much greaterconfidence and specificity than “downstream” sec-tors (those that depend on them). Infrastructureorganizations thus appear to be lacking a compre-hensive view of how a disaster is likely to affect allthe various interdependent infrastructure sectors.

4.1. Assessment of Service Disruption Levels

To help address this gap, we synthesized infor-mation from the interviews into an initial diagram ofsectoral disruption levels and presented it to infras-tructure providers at the workshop for review, revi-sion, and feedback. We also discussed the experiencein other similar events as a means of avoiding over-confidence in assessments. We sought informationon the perspectives of other sectors regarding theirviews on a given sector’s expected function. Partici-

pants had the opportunity to revise their own sector’sservice disruption rating after taking into accountother sectors upon which they rely. A number of sec-tors chose to change their ratings and in many casesraised their expected disruption level, thus provid-ing a more informed assessment of expected service-level disruptions. Fig. 4 represents the final diagramof service disruption levels after revisions were madeby workshop participants in the 2008 workshop.10

Participants commented that although the diagramwas helpful in understanding what to expect fromother infrastructure providers, there were concernswith the specific accuracy of aggregating across dif-ferent agencies within sectors. Another concern wasthe aggregation across the spatial dimension, in thatsome areas of the Lower Mainland would face muchmore serious infrastructure disruptions while oth-ers may remain relatively undamaged. (In the inter-views, however, they had been unable or unwilling toprovide information on how service disruption mightvary across the study region.)

Fig. 4 can be interpreted as a regional, multisec-toral version of the conceptual resilience diagram(Fig. 1) for a potential earthquake event in the MetroVancouver region. It shows that all sectors exceptgovernment and natural gas are expected to experi-ence severe service disruptions in the immediate af-termath of the specified earthquake scenario. Causesrange from physical damage of hard infrastructure(such as roads, bridges, and power distribution sys-tems) to overwhelmed telecommunication systems.After 72 hours, all infrastructure sectors expect to beexperiencing moderate disruption. After two weeks,electric power, government, and the natural gassectors expect to have recovered to low service dis-ruptions. No infrastructure sector is expected to havecompletely returned to no loss service disruptionlevels at this stage. Interestingly, some infrastructureproviders commented that after the initial time

10Note that after our 2009 workshop, discussion and revisions tothe diagrams continued for over two months, based on evolv-ing understanding of potential interactions. In the workshop,two significant failure modes for one major infrastructure sec-tor were identified that had not been originally reflected in thefirst round of interviews. These failure modes had to do with theimpacts of a dyke breach and the depth of flooding in our floodscenario. These two potential failure modes could adversely af-fect two of the most significant infrastructure sectors in the re-gion, in ways that had not been considered previously. We thenconsulted existing engineering studies to clarify that these failuremodes were indeed possible, and highly likely with floods over agiven level, based on the dyke breach scenario used in our 2009work.

Page 11: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

426 Chang et al.

Fig. 4. Estimated service disruption levels, M7.3 earthquake scenario.

period, service levels might actually deteriorate dueto the effects of service losses in upstream infras-tructure systems and depletion of backup resources;however, they did not go so far as to revise thediagram to reflect this possibility.

4.2. Interdependency Diagrams

Diagrams were also developed to overlay the ex-pected disruption levels in each sector with infor-mation on interdependencies between sectors. Fig. 5provides a comprehensive overview of the upstreamand downstream effects in the immediate aftermathof the earthquake. The dependencies are indicatedby links or arrows that point in the direction of thedownstream sector. Line widths of the links indicatethe degree of functional dependency. Colors of thelinks and ovals refer to service disruption levels (e.g.,red denotes “severe” service disruption and yellowsignifies “moderate” disruption) consistent with theclassification scheme in Figs. 3 and 4. Service disrup-tion information, obtained from the interviews, ac-counts for upstream dependencies and backup sys-tems that may already be in place. Fig. 5 reveals thestatus of upstream sectors, allowing each sector torevise its expectations of the likelihood of receiv-ing materials or service from other sectors, and toplan and implement measures accordingly (e.g., ac-quire backups, stockpile resources, design redundan-cies). For example, one of the links indicates that

the health-care sector is highly dependent on wa-ter, which in turn is expected to experience severeservice disruption. This knowledge helps the health-care sector make decisions regarding alternative op-tions, such as constructing wells onsite, to supple-ment potable water sources.

Fig. 5 indicates that some sectors are much moreextensively linked than others. Core and peripheralsectors can be distinguished by the number of down-stream dependencies, weighted by the strength of thedependencies. In this sense, the data indicate thatelectric power is the most connected infrastructure,followed by land transportation and telecommunica-tions, in that order. These sectors can be consideredthe region’s core infrastructures.

It is interesting to compare Fig. 5 for MetroVancouver with the generalized interdependenciesdiagram from Rinaldi et al.,(50) which provided itsconceptual framework. From a conceptual stand-point, Rinaldi et al.(50) suggest that electric powerand telecommunications are interdependent with allother infrastructures, that transportation is interde-pendent with most others, and that natural gas ismore peripheral. This study provides empirical con-firmation for these suggestions in the case of MetroVancouver. A key difference relates to water: bothRinaldi et al.(50) and experience in major disasters(13)

indicate that most other infrastructures aredependent on water, yet this was not found inthe Vancouver study. Further study is required to

Page 12: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 427

Fig. 5. Infrastructure interdependencies and service disruptions.Note: Sea and air transportation have not been represented on this diagram due to less perceived interactions with the other sectors. Themain downstream dependency for the Vancouver Port is to ensure online communication, while the airports have a significant dependencyon ground transportation, and moderate dependencies on electric power, telecommunications, and water.

investigate whether this reflects an actual or simply aperceived lack of reliance on water.

4.3. Workshop Discussion

The study approach culminated in a workshopthat provided a number of benefits for character-izing and addressing the region’s infrastructurevulnerability and resilience. Participants gained amore comprehensive and complex understanding ofinfrastructure interdependencies and their potentialoutcomes in disasters. In several cases, participantsrevised judgments they had provided in the inter-views after considering and discussing the expec-tations of service disruption in other infrastructuresectors. Typically, participants increased the degreeof dependency on other upstream sectors once theyrealized the potential levels of service disruption.Discussion also revealed that upstream service losses

would likely be exacerbated in the days and weeksfollowing the disaster as a result of backup resourcesbecoming depleted. Participants also became moreaware of how sectors directly upstream were in turndependent upon other sectors in a complex chain ofinterdependencies. The diagrams also precipitateddiscussion of facets they could not capture, includingvariation in impacts across urban space and specificfacilities. Participants recognized additional inter-dependencies that they had not considered at theindividual interviews; for example, the dependenceof other infrastructures on the health-care sectorfor medical treatment for staff and their families.Finally, discussions allowed resolution of somediscrepancies in expectations that had been revealedin the interviews. Telecommunications, for example,had assumed that electric power distribution wouldbe devastated in a major earthquake, whereasthe electric power sector in fact did not expect

Page 13: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

428 Chang et al.

substantial system damage. The health-care sectorexpected that local roads would be damaged butlargely usable in a seismic event. The land transporta-tion sector, however, expected traffic to be unableto move in many areas. By opening the discussionon these types of discrepancies in expectations, theprocess allowed infrastructure providers to developcommon understandings across sectors, as well as asounder basis for making planning decisions.

4.4. Validation

In the realm of quantitative modeling, verifica-tion of data and validation of results are importantnorms. When attempting to model a complex phe-nomenon based on expert judgments, there are alsonorms for verification and validation, but they arenaturally somewhat different. Earlier, we stressedthat both quantitative and qualitative modeling relyheavily on judgments. The approach outlined hereseeks to make those judgments more explicit andopen to discussion and revision. Expert elicitationassumes only that (i) it is possible to characterizeuncertainty as degrees of belief in specific judgments,which is a fundamental precept of Bayesian statistics,and (ii) the views of the best-informed expertsavailable are a reasonable source of information,when structured and elicited appropriately.(19) Theseassumptions are minimal compared to those neededto make use of, say, quantitative input-outputapproaches to characterizing impacts of extremeevents.(53,54) Hence, the requirements for validationare markedly different.

For obvious reasons, expert judgments regardingpatterns of consequences from extreme events can-not be validated through experience until an eventhas occurred. Yet that does not mean these judg-ments are without content or value. The value of suchjudgments is in part a function of the decision con-text, or how the information is intended for use. Fordecades, decision analysts have made use of elicitedjudgments as a basis for comparing the performanceof alternatives, using specific sets of beliefs (proba-bilities) and values (tradeoffs).(46) In other contexts,probability elicitation has been used almost as a formof traditional science consolidation and augmenta-tion, to characterize outcomes (say, given future cli-mate change scenarios) even if no direct mitigationor adaptation options exist.(55,56)

Given that one cannot validate these findingsin terms of outcomes, it is nevertheless importantto consider validation in terms of the process em-

ployed to obtain the judgments.11 The process em-ployed to elicit the judgments in this work, summa-rized in Fig. 2, has been informed by three sources.One is the writing on the design and practice ofprobability elicitation, noted earlier.(19,47) A secondsource is the writing on steps to improving judg-mental forecasting.(58,59) A third is writing on theDelphi (i.e., consensus-based) approach to elicitingexpert judgments.(60) From these sources, the ap-proach in Fig. 2 was developed as our overall strat-egy. The method is iterative, as is the Delphi ap-proach, but does not attempt to force any consensus.

One approach used in probability elicitation forvalidation of judgments is convergence, achieved byasking the same basic question in several differentways. Here we instead asked the questions multipletimes, first in the interview process, and then subse-quently in the workshop process after learning aboutthe initial results and the views of others. A secondapproach in probability elicitation is to ensure thatall terms are clearly defined and all participants havethe same scenarios in mind when providing the judg-ments. We addressed these points through direct in-terview processes in which we could explain all termsand questions, and with a clearly defined earthquakescenario. A third precept of probability elicitation isto ensure all participants are familiar with key lit-erature, and that they use all available informationto inform the judgments. We distributed informationbefore the interviews and workshop on the effectsof five previous earthquakes in various urban areasaround the world. Later, when we reconsidered theinitial judgments in the workshop process, we madereference to these previous events once again to en-sure the information was salient for the participants.

Some steps could be taken in the future to im-prove the validity of the judgments in subsequentapplications of this work. One obvious step wouldbe to elicit judgments from more experts.12 Onereason we did not adopt that approach here is

11This reasoning is in keeping with approaches to evaluating thequality of decisions based not on outcomes, but rather on theprocess, structure, and information base.(57)

12If multiple experts were employed in future studies, additionalsteps could be taken to evaluate and assess the quality of judg-ments for such a process. For example, in a previous study(56)

we omitted some expert views because they were highly incom-plete, and in one case indicated misunderstanding of the ques-tions. One can create histograms of sets of judgments for a givenvariable, to serve as a basis for considering strategies for com-bining experts. Beyond these steps, one can draw insights bycomparing the views of groups of experts (say with differentbackgrounds).(19)

Page 14: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 429

because this work is deliberately extensive (all themajor infrastructure systems in a region) rather thanintensive (with several experts considering one sys-tem in a more constrained location). The argumentin favor of interviewing multiple experts is simplythat in contexts with extreme uncertainty, no oneview about the future is likely to be accurate.(58) Yet,in almost every situation, including this one, obtain-ing more expert views is far easier said than done.We sought the views of infrastructure owners aboutwhom they wanted to provide the judgments regard-ing the performance of their system in a specificearthquake scenario. The individuals they selectedare those whom the organizations trust to providethis information. There are no alternative experts fa-miliar with these specific systems.

A second way to validate the judgments pro-vided here would be to compare these judgments tothe actual experience with infrastructure systems inprevious major earthquakes. Although that approachhas appeal, it is of limited value here because we al-ready provided data on actual impacts for five majorearthquakes to the participants as part of the back-ground preparation for the survey and workshop.Moreover, the vulnerability of infrastructure systemsdepends on local conditions (e.g., siting with respectto topographic, soil, and other geographic conditions;vintage of physical components; design and networkconfiguration) and earthquake damage depends onevent characteristics (e.g., epicentral location, depth,magnitude), so that detailed validation using expe-rience data from other events and cities remainschallenging.

Yet another perspective on validation can be dis-cerned from the writing on confidence in expert judg-ment, which indicates that experts (like all people)tend to be overconfident in judgments, leading to toolittle recognition of the nature and likelihood of ex-treme events, and thus too narrow a representationof uncertainty.(48,61) In that vein, one should expectthat the estimates in Fig. 4 for duration and disrup-tion are more likely to be underestimates than over-estimates. Beyond that, the challenges of judgmentsabout dynamics of complex systems are formidablewhen the potential for threshold effects and emer-gent surprises are recognized. However, our focushere on failure interactions among systems is a steptoward understanding and representing these thresh-olds, surprises, and complexities.

A final issue is that validation should reflect thepurpose of doing the analysis. Here our purpose inthis work is not to evaluate any single alternative

or sets of alternatives to enhance regional resilience.There are literally tens of millions of alternatives thatcould be considered, which need to be analyzed atmarkedly different levels than in terms of the per-formance of all infrastructure systems in the regionafter a specific extreme event. Our purpose is not toprovide the “right” estimates of the consequences ofan earthquake like that in our scenario, since thesecannot be verified before the fact. Our purpose is in-stead to develop and apply an approach that couldtackle the informational market failure we outlinedin the first section of this article. The intent is to helpinform regional managers for individual systems, andmanagers for society as a whole, about the infrastruc-ture interdependencies within the region, how theycould be affected by an extreme event such as a ma-jor earthquake, and the potential consequences forsociety. Regional planners and infrastructure ownerscould greatly benefit from the information summa-rized in Figs. 4 and 5; we know of no other ways toobtain such information.

5. CONCLUSIONS

Understanding infrastructure vulnerability andresilience is essential to the planning and decisionmaking that can enhance the disaster resilience ofcities and their vital infrastructure systems. This re-search has provided a new, practical method for char-acterizing infrastructure resilience, emphasizing thecritical but often overlooked factor of IFIs, and ap-plied it to the Metro Vancouver study region. It has,moreover, demonstrated a process that we believecan be applied by other communities to characterizeand address their infrastructure vulnerabilities.

The approach outlined here provides a meansto help overcome the informational market failuresof partial incentives, limited and asymmetric infor-mation, and lack of experience. To our knowledge,this is the first method available to characterize theresilience of a community or region to a given ex-treme event scenario, in terms of all its componentinfrastructure systems, reflecting the potential for in-terdependent failures in these systems. It builds onprobabilistic elicitation approaches, but with severalkey differences, primarily the direct elicitation ofresilience (robustness and recovery) rather thanprobability of a given outcome, within a series ofsteps that includes interaction among different in-frastructure systems to provide learning and the po-tential for revised estimates. This represents an ex-tension and elaboration of established elicitation

Page 15: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

430 Chang et al.

methods, within a new application context of greatimportance for risk management efforts and forsocieties.

Findings indicated the need for greater infor-mation sharing among infrastructure organizationsregarding the likely effects of a major disaster.Specific results for the case study pertain to thelikely severity of infrastructure disruptions in anearthquake scenario, with severity defined in termsof the impact intensity and spatial extent of disrup-tion. For the hypothetical M7.3 earthquake, severeinfrastructure service disruptions can be expectedin most infrastructure sectors in the immediateaftermath. Economic, social, and health-relatedimpacts are likely to be severe immediately after theevent, particularly in locations where ground shakingis expected to be most pronounced. Emergencyrestoration efforts are anticipated to reduce servicedisruptions to moderate levels within 72 hours, butfull restoration would require more than two weeks.Results also indicate that government and naturalgas are expected to be more resilient in such an eventthan other infrastructure sectors.

In addition to developing information on re-silience, the overall approach also fosters the ca-pacity of the infrastructure organizations to addressrisk. The workshop format allowed infrastructuremanagers to share information on vulnerability, re-silience, and interdependencies in a structured andefficient manner in order to develop a communityor regional perspective of infrastructure resiliencebeyond their own systems. As noted, participantsalso gained new understanding on IFIs that allowedthem to revise their expectations of their own in-frastructure system’s resilience. Although this over-all approach cannot substitute for detailed, long-termplanning within infrastructure organizations, it canprovide new knowledge and networks from a re-gional perspective that help motivate, inform, andsupport such planning.

A number of limitations should be noted regard-ing the methodology developed here. The approachdeveloped one specific earthquake scenario (andin the subsequent year of the study, one floodscenario) in order to focus participants’ attentionon a common context. Consequently, variations ineffects across other potential earthquakes and othertypes of hazards were not discussed. In addition, theapproach emphasized regional-scale characteriza-tions of infrastructure interdependencies and serviceprovision. It does not delve into the technical, largely

engineering specifics of these interdependencies.Further, in most cases, we interviewed only onerepresentative of each organization; while the mostknowledgeable expert, this person may not alwayshave a complete understanding of the organization’sactivities.

It is also important to recognize how this workshould be interpreted and employed. Like all ap-plications of expert elicitation, the process outlinedhere is not meant to eliminate the need for quanti-tative analysis of major systemic failures in extremeevents. We do not expect that experts engaged insuch a process will be able to capture all the com-plexities, interdependencies, and related emergentbehaviors of such critical infrastructures. Rather, thejudgment-based analysis outlined here is a funda-mental initial step in identifying vulnerabilities andcritical interdependencies. It also provides structuredinformation and knowledge needed to conduct fur-ther quantitative analyses. The work of Kroger(62) ishelpful in showing what kinds of quantitative analy-sis could contribute to identifying robust alternatives.But as Kroger notes, the state-of-the-art of model-ing interdependencies is still far from routinely op-erational. Hence it is important to tackle such as-sessments in a stepwise and integrated fashion. Theanalysis discussed here is an important first step inmore in-depth analysis of critical infrastructure inter-dependencies in extreme events.

Overall, the study has developed and demon-strated a practical approach that can be readily ap-plied by disaster managers to reduce risks associ-ated with IFIs in their communities.13 The problemof IFIs is challenging from both a technical stand-point and a planning perspective: there are disjunc-tures between infrastructures’ private interests andthe broader societal interests, barriers to informationsharing, and few opportunities for learning from ex-perience. To address these challenges, the approachdeveloped here—consisting of a linked sequenceof informational summaries, a regional disaster sce-nario, expert interviews, synthesis diagrams, andinteractive workshops—emphasizes both develop-ing information on regional risk and engaging thespectrum of infrastructure organizations in thisprocess.

13We have documented outcomes of the research on the projectwebsite (www.chs.ubc.ca/dprc koa/) in formats appropriate topractitioner audiences, including a searchable database andpractitioner-oriented short reports.

Page 16: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 431

Further research is needed to validate and refinethe approach developed here. In particular, methodsare needed that can efficiently and effectively elicitexpert judgments regarding a range of potential haz-ard events that can cause a range of infrastructuredisruptions and interdependencies. Experience fromour 2009 flood application suggests that one promis-ing approach may be to fully develop a single sce-nario and from there, to explore variations (e.g., inflood depth, duration, location, timing) that couldtrigger significant threshold effects. For example, aparticular infrastructure may only fail if flood lev-els exceed a certain depth. Validation of expert elic-itation for rare events is not possible with standardmethods. On the other hand, validation may be pos-sible in the future through expanded databases of theconsequences of extreme events such as earthquakesfor comparable regions, and expanded databases ofIFI events, in order to compare elicited judgments topatterns in actual events. Calibration of the qualityof judgments for a given expert may be possible inthe future, if one adopts the assumption that calibra-tion in one domain translates into calibration in otherdomains.(23)

Further research is also needed on developing,analyzing, and implementing alternatives to increaseregional resilience to IFIs. The scope and number ofalternatives is enormous, and the tasks of comparingand setting priorities among these alternatives meritssubstantial further research.(63,64)

Our experience in conducting this work alsopoints to a major gap in governance for disastermanagement in terms of infrastructures. Althoughemergency response is a standard activity for gov-ernance contexts, and efforts are underway to helpcoordinate security measures for facilities in variouslocations, we know of no governance context thataddresses issues of infrastructure interdependenciesand their implications for regional resilience. Hence,there seems to be a role for new (formal or infor-mal) governance efforts that involve coordination ofefforts and information sharing as discussed in thisarticle, and are also provided with mandates to re-duce such risk. We hope this work spurs interest indeveloping such governance mechanisms.

ACKNOWLEDGMENTS

We would like to thank the interview and work-shop participants for their time and participation inthis study, as well as the BC Provincial Emergency

Program for sharing its earthquake scenario for theproject. We also thank the anonymous reviewersof this article for helpful comments. Research as-sistants David Hawkins, Andrea Procyk, CourtneyBeaubien, and Gerard Chew also contributed tothis project. This research was supported by Infras-tructure Canada through the Knowledge, Outreachand Awareness Program, and the National ScienceFoundation under grant number CMS-0332002. Theefforts of Tim McDaniels in preparing the articlewere supported by the Climate and Energy Decision-Making Center (CEDM) located in the Departmentof Engineering and Public Policy, through a cooper-ative agreement between the National Science Foun-dation (SES-0949710) and Carnegie Mellon Univer-sity. The CEDM in turn supports researchers in theInstitute for Resources, Environment and Sustain-ability at the University of British Columbia.

APPENDIX: INTERVIEW AND WORKSHOPMATERIALS

This appendix includes selected materials to helpdocument the methods described in this study, specif-ically an interview script used in the flood inter-views (i.e., a refinement of the interview script usedin the earthquake study), including associated judg-ment elicitation tables.

INTERVIEW SCRIPT

Part 1: Introduction to Research and Scenarios

Please read over the attached scenario regard-ing a flood in the Chilliwack region and refer tothe accompanying map. Please note that the in-frastructure that is affected by this flood servicesthe Greater Vancouver Regional District (GVRD).As such, we are most interested in the effects onGreater [Metro] Vancouver. The purpose of thisscenario is to provide a specific focus for yourthinking about your sector given a major systemshock.

(1) Does your organization use a flood scenariofor planning, and if so, how does it differ fromthis one?

(2) Do you have any concerns or comments aboutthe scenario? Is there anything you wouldmodify or add?

Page 17: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

432 Chang et al.

Part 2: Sector Impact

(3) Please use Table I to indicate how the floodscenario would affect your ability to provideservices to consumers at various time points.Please consider only how the flood would af-fect your system (i.e., assume that other in-frastructures do not suffer any damage ordisruption). What is your rough estimate ofthe total duration of service loss? What doyou expect to be the severity of servicedisruption?

Table I14

Service Disruptions to YourSystem (Nature and Severity)

Scenario 0 Hours 72 Hours 2 Weeks+

Flood � No loss �No loss �No loss� Slight

disruption� Slight

disruption� Slight

disruption�Moderate

disruption�Moderate

disruption�Moderate

disruption� Severe

disruption�Severe

disruption� Severe

disruptionDescription: Description: Description:

Part 3: Upstream IFIs

(4) To what extent are you dependent on otherinfrastructures? Please consider your depen-dency in the postdisaster time frame. Whenanswering, please consider your degree of re-liance on these infrastructures, as well as yourimplemented flood-related mitigations to re-duce this reliance. Please use Table II for yourresponses.

14The interviewer guidelines included additional prompts and clar-ifications for use when needed, including for Table I: “Explainthe types of service disruptions to your sector for each of thethree times. Please consider any measure of service loss that isappropriate to your system.” Further, the service disruption lev-els were defined as follows: Slight = Disruptions with low spatialextent and low impact; Moderate = Disruptions with low spa-tial extent & high impact, OR high spatial extent & low impact;Severe = Disruptions with high spatial extent & high impact dis-ruptions.

Table II15

Extent of Dependence on Sector

Sector None Slight Moderate Significant Do not know

PowerCommunicationWaterTransportationGovernmentNatural gasWastewaterHealth systemOther

(5) How do you expect other infrastructures tobe affected by the flood scenario? Please useTable III for your responses.

Table III

Type of Disruption to Sector

Sector None Slight Moderate Severe Do not know

PowerCommunicationWaterTransportationGovernmentNatural gasWastewaterHealth systemOther

(6) With regards to potential upstream disrup-tions, are there any specific concerns you mayhave about certain sectors?

(7) Now considering how you expect these otherinfrastructures to be disrupted, how would theflood affect your system?

After participant has completed both tables:

(8) Now consider if the flood were smaller orlarger than in the scenario. How would youranswers change? What variables regardingyour system are most affected by the severityof the flood?

15The interviewer guidelines defined dependence levels as: Slight= Minimal reliance on sector; Moderate = Large reliance on sec-tor with significant backup available, or, moderate reliance onsector with no backup available; Significant = Large reliance onsector with limited backup available.

Page 18: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Toward Disaster-Resilient Cities 433

Wrap Up

(9) What information sources influence yourviews on these effects?

(10) Is there any important information that wefailed to cover in our interview? Is there any-thing else that you would like to add?

(11) Are there any people within or not withinyour organization that you recommend thatwe talk to?

(12) Is there any information that you shared withus today that you would like us NOT to sharewith others at our upcoming workshop?

(13) Overall, how confident are you in your an-swers? Please fill out Table IV with a checkmark for your level of confidence.

Table IV

1 2 3 4 5(Not at All (HighlyConfident) (Confident) Confident)

Your systemUpstream

sectorsEarthquake-

relatedquestions

Flood-relatedquestions

Finally, we plan to send all participants a briefsummary of their interview. We would welcome yourcomments on this report.

Thank you very much for your participation inthis interview!

REFERENCES

1. Mileti DS. Disasters by Design: A Reassessment of NaturalHazards in the US. Washington, DC: National Academy Press,1999.

2. Burby RJ, Beatley T, Berke PR, Deyle RE, French SP, God-schalk DR, Kaiser EJ, Kartez JD, May PJ, Olshansky R, Pa-terson RG, Platt RH. Unleashing the power of planning in cre-ating disaster-resilient communities. Journal of the AmericanPlanning Association, 1999; 65(3):247–258.

3. Godschalk DR. Urban hazard mitigation: Creating resilientcities. Natural Hazards Review, 2003; 4(3):136–143.

4. United Nations International Strategy for Disaster Reduction(UN/ISDR). Hyogo Framework for Action 2005–2015: Build-ing the Resilience of Nations and Communities to Disasters.Geneva, Report No.: UN/ISDR-07-2007-Geneva.

5. Berke PR, Campanella TJ. Planning for postdisaster re-siliency. Annals of the American Academy of Political and So-cial Science, 2006; 604(1):192–207.

6. Burby RJ. Cooperating with Nature: Confronting NaturalHazards with Land Use Planning for Sustainable Communi-ties. Washington, DC: Joseph Henry Press, 1998.

7. Burby RJ. Plans, code enforcement, and damage reduction:Evidence from the Northridge earthquake. Earthquake Spec-tra, 1998; 14(1):59–74.

8. Nelson AC, French SP. Plan quality and mitigating damagefrom natural disasters: A case study of the Northridge earth-quake with planning policy considerations. Journal of theAmerican Planning Association, 2002; 68(2):194–207.

9. Brody SD, Zahran S, Maghelal P, Grover H, Highfield WE.The rising costs of floods: Examining the impact of plan-ning and development decisions on property damage inFlorida. Journal of the American Planning Association, 2007;73(3):330–345.

10. Schwab J. Planning for Post-Disaster Recovery and Recon-struction. Chicago, IL: American Planning Association, 1998.

11. Olshansky RB. Planning after Hurricane Katrina. Journal ofthe American Planning Association, 2006; 72(2):147–153.

12. Gordon P, Richardson HW, Davis B. Transport-related im-pacts of the Northridge earthquake. Journal of Transportationand Statistics, 1998; 1(2):21–36.

13. Nojima N, Kameda H. Lifeline interactions in the Hanshin-Awaji earthquake disaster. Pp. 253–264 in Hamada H,Ohmachi T, Ohbo N (eds). The 1995 Hyogoken-Nanbu Earth-quake: Investigation into Damage to Civil Engineering Struc-tures. Tokyo: Japan Society of Civil Engineers, 1996.

14. Mendonca D, Wallace WA. Impacts of the 2001 World TradeCenter attack on New York City critical infrastructures. Jour-nal of Infrastructure Systems, 2006; 12(4):260–270.

15. Chang SE, McDaniels TL, Mikawoz J, Reed D. Infrastructurefailure interactions in extreme events: The 1998 ice storm. Nat-ural Hazards, 2006; 41(2):337–358.

16. McDaniels T, Chang S, Peterson K, Mikawoz J, Reed D.Empirical framework for characterizing infrastructure failureinterdependencies. Journal of Infrastructure Systems, 2007;13(3):175–184.

17. Keeney RL, von Winterfeldt D. On the uses of expert judg-ment on complex technical problems. IEEE Transactions onEngineering Management, 1989; 36(2):83–86.

18. Keeney RL, von Winterfeldt D. Eliciting probabilities fromexperts in complex technical problems. IEEE Transactions onEngineering Management, 1991; 38(3):191–201.

19. Morgan MG, Henrion M, Small M. Uncertainty: A Guideto Dealing with Uncertainty in Quantitative Risk and PolicyAnalysis. New York: Cambridge University Press, 1990.

20. Innes JE, Booher DE. Consensus building and complex adap-tive systems: A framework for evaluating collaborative plan-ning. Journal of the American Planning Association, 1999;65(4):412–423.

21. McDaniels TL, Gregory RS, Fields D. Democratizing riskmanagement: Successful public involvement in local watermanagement decisions. Risk Analysis, 1999; 19(3):497–510.

22. Kaplan S, Garrick BJ. On the quantitative definition of risk.Risk Analysis, 1981; 1(1):11–27.

23. Cooke R. Experts in Uncertainty: Opinion and SubjectiveProbability in Science. New York: Oxford University Press,1991.

24. Carpenter S, Walker B, Anderies JM, Abel NA. Frommetaphor to measurement: Resilience of what to what?Ecosystems, 2001; 4(8):765–781.

25. Holling CS. Engineering resilience versus ecological re-silience. Pp. 31–44 in Schulze P (ed). Engineering WithinEcological Constraints. Washington, DC: National AcademyPress, 1996.

26. Bruneau M, Chang SE, Eguchi RT, Lee GC, O’Rourke TD,Reinhorn AM, Shinozuka M, Tierney K, Wallace WA, vonWinterfeldt D. A framework to quantitatively assess and en-hance the seismic resilience of communities. Earthquake Spec-tra, 2003; 19(4):733–752.

Page 19: Toward Disaster-Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

434 Chang et al.

27. Chang SE, Shinozuka M. Measuring improvements in the dis-aster resilience of communities. Earthquake Spectra, 2004;20(3):739–755.

28. Morgan K. The learning region: Institutions, innovation andregional renewal. Regional Studies, 1997; 31(5):491–503.

29. Lebel L, Anderies JM, Campbell B, Folke C, Hatfield-DoddsS, Hughes TP, Wilson J. Governance and the capacity to man-age resilience in regional social-ecological systems. Ecologyand Society, 2006; 11(1):1–19.

30. Simpson DM, Rockaway TD, Weigel TA, Coomes PA, Hol-loman CO. Framing a new approach to critical infrastructuremodelling and extreme events. International Journal of Criti-cal Infrastructure, 2005; 1(2/3):125–143.

31. Rose A, Liao SY. Modeling regional economic resilience todisasters: A computable general equilibrium analysis of wa-ter service disruptions. Journal of Regional Science, 2005;45(1):75–112.

32. Kajitani Y, Tatano H. Estimation of lifeline resilience factorsbased on surveys of Japanese industries. Earthquake Spectra,2009; 25(4):755–776.

33. Duenas-Osorio L, Craig JI, Goodno BJ. Seismic response ofcritical interdependent networks. Earthquake Engineering &Structural Dynamics, 2007; 36(2):285–306.

34. Min HSJ, Beyeler W, Brown T, Son YJ, Jones AT. Towardmodeling and simulation of critical national infrastructure in-terdependencies. IIE Transactions, 2007; 39(1):57–71.

35. Haime YY, Jiang P. Leontief-based model of risk in complexinterconnected infrastructures. Journal of Infrastructure Sys-tem, 2001; 7(1):1–12.

36. Apostolakis GE, Lemon DM. A screening methodology forthe identification and ranking of infrastructure vulnerabilitiesdue to terrorism. Risk Analysis, 2005; 25(2):361–376.

37. Lian CY, Santos JR, Haimes YY. Extreme risk analysis of in-terdependent economic and infrastructure sectors. Risk Anal-ysis, 2007; 27(4):1053–1064.

38. McNally RK, Lee SW, Xiang WN. Learning the critical infras-tructure interdependencies through an ontology-based infor-mation system. Environment and Planning B-Planning & De-sign, 2007; 34(6):1103–1124.

39. Brown C, Chang SE, McDaniels T. Utility provider liabilityfor electrical failure: Implications for interdependent criticalinfrastructure. Electricity Journal, 2006; 19(5):69–81.

40. Chang SE. Evaluating disaster mitigations: Methodology forurban infrastructure systems. Natural Hazards Reviews, 2003;4(4):186–196.

41. Feinberg LE. FOIA, federal information policy, and informa-tion availability in a post-9/11 world. Government InformationQuarterly, 2004; 21(4):439–460.

42. Hammitt H. Less safe: The dismantling of public informationsystems after 9/11. Social Science Computer Review, 2005;23(4):429–438.

43. Fiorino DJ. Citizen participation and environmental risk: Asurvey of institutional mechanisms. Science, Technology, &Human Values, 1990; 15(2):226–243.

44. Gregory R, Failing L, Ohlson D, McDaniels TL. Some pitfallsof an overemphasis on science in environmental risk manage-ment decisions. Journal of Risk Research, 2006; 9(7):717–735.

45. Slovic P. Trust, emotion, sex, politics, and science: Surveyingthe risk-assessment battlefield (reprinted from Environment,Ethics, and Behavior, pp. 277–313, 1997). Risk Analysis, 1999;19(4):689–701.

46. Keeney RL. Value-Focused Thinking: A Path to Creative De-cision Making. Cambridge, MA: Harvard University Press,1992.

47. MacNamee P, Celona J. In Celona J (ed). Decision Analysisfor the Professional with Supertree. Menlo Park, CA: Scien-tific Press, 1987.

48. Kahneman D, Slovic P, Tversky A. Judgment Under Un-certainty: Heuristics and Biases. Cambridge, UK: CambridgeUniversity Press, 1982.

49. Analyzing Infrastructures for Disaster-Resilient Communi-ties. Available at: http://www.chs.ubc.ca/dprc˙koa/, AccessedSeptember 2013.

50. Rinaldi SA, Peerenboom JP, Kelly TK. Identifying, under-standing, and analyzing critical infrastructure interdependen-cies. IEEE Control Systems Magazine, 2001; 21(6):11–25.

51. Chang SE, McDaniels TL, Reed D. Mitigation of extremeevent risks: Electric power outage and infrastructure failureinteractions. Pp. 70–90 in Richardson HW, Gordon P, MooreJE II (eds). The Economic Impacts of Terrorist Attacks. Chel-tenham, UK: Edward Elgar, 2005.

52. Cox LA. What’s wrong with risk matrices? Risk Analysis,2008; 28(2):497–512.

53. Haimes YY, Horowitz BM, Lambert JH, Santos JR, Lian C,Crowther KG. Inoperability input-output model for interde-pendent infrastructure sectors. I: Theory and methodology.Journal of Infrastructure Systems, 2005; 11(2):67–79.

54. Santos JR, Haimes YY. Modeling the demand reductioninput-output (I-O) inoperability due to terrorism of inter-connected infrastructures. Risk Analysis, 2004; 24(6):1437–1451.

55. Zickfeld K, Levermann A, Morgan MG, Kuhlbrodt T, Rahm-storf S, Keith DW. Expert judgements on the response of theAtlantic Meridional overturning circulation to climate change.Climatic Change, 2007; 82(3):235–265.

56. McDaniels T, Mills T, Ohlson D, Gregory R. Exploring robustalternatives for climate adaptation in forest-land managementthrough expert judgments. Risk Analysis, 2012; 32(12):2098–2112.

57. Arvai JL, Froschauer A. Good decisions, bad decisions: Theinteraction of process and outcome in evaluations of de-cision quality. Journal of Risk Research, 2010; 13(7):845–859.

58. Stewart TR, Lusk CM. 7 components of judgmental fore-casting skill: Implications for research and the improve-ment of forecasts. Journal of Forecasting, 1994; 13(7):579–599.

59. Armstrong JS. Principles of Forecasting: A Handbook for Re-searchers and Practitioners. Boston, MA: Kluwer Academic,2001.

60. Linstone HA, Turoff M. The Delphi Method: Techniques andApplications. Reading, MA: Addison-Wesley Pub. Co., Ad-vanced Book Program, 1975.

61. Soll JB, Klayman J. Overconfidence in interval estimates.Journal of Experimental Psychology—Learning Memory andCognition, 2004; 30(2):299–314.

62. Kroeger W. Critical infrastructures at risk: A need for anew conceptual approach and extended analytical tools. Re-liability Engineering & System Safety, 2008; 93(12):1781–1787.

63. Procyk A. Creating Alternatives for Improving Infras-tructure Interdependency Resilience, 2010. Available at:http://www.chs.ubc.ca/dprc koa/pdf files/PR No3 Creating%20Alternatives.pdf, Accessed August 2013.

64. Procyk A. Setting Priorities for Improving Infrastructure In-terdependency Resilience, 2010. Available at: http://www.chs.ubc.ca/dprc koa/pdf files/PR No4 Setting%20Priorities.pdf,Accessed August 2013.