Conceptualizing and Measuring Immigration Policies. A Comparative Perspective

Marc Helbling, Liv Bjerre, Friederike Römer and Malisa Zobel

Conceptualizing and Measuring Immigration Policies.

A Comparative Perspective

Liv Bjerre, Marc Helbling, Friederike Römer and Malisa Zobel

WZB Berlin Social Science Center

Abstract

In the last decade, researchers have developed many innovative ideas for the construction of

indices measuring immigration policies. Methodological considerations have, however, been

largely absent from the discussion. To close this gap, this paper investigates the characteristics

of existing indices by critically comparing and discussing them. We start by providing a defi-

nition of immigration policy which may serve as a benchmark when assessing the indices. By

means of the analytical framework developed by Munck and Verkuilen (2002), which we

adapt and customize for our analysis, we then evaluate the conceptualization, measurement,

and aggregation, as well as the empirical scope of thirteen immigration policy indices. We

conclude by discussing methodological strengths and weaknesses of the indices, how these

affect the research questions that can be answered and what the next steps in index building

within the field of immigration policy should be.

(Work in progress, November 2013)

2

Introduction

Immigration and its regulation have become a salient political issue in all industrialized coun-

tries. This has also affected the social sciences, and increasingly researchers are now turning

to comparative approaches, aiming at going beyond single case studies and small-N analyses.

The growing interest in comparative analyses of immigration has led to a variety of attempts

to quantify immigration policy and to build indices.1

This paper proposes to investigate the strengths and weaknesses of existing indices by

critically comparing and discussing them. We adapt and slightly modify a framework of com-

parison developed by Munck und Verkuilen (2002), which was initially designed to compare

democracy measures. They identify three sequential and interrelated stages of the index-

building process that need to be evaluated: (I) Conceptualization, (II) Measurement, and (III)

Aggregation, each of which has a number of corresponding substeps. Munck and Verkuilen’s

(2002) three-step assessment process has turned out to be very useful for assessing policy

indices in fields as different as democracy (Müller and Pickel 2007) or state-church regimes

(Traunmüller 2012).2 Since we believe that a sound methodology of index building should be

used across thematic fields and disciplines we chose the framework of Munck and Verkuilen.

It seems that the pioneering spirit of index building in the social sciences has left little

space for methodological considerations: The reviews of indices in various fields deplore their

methodological shortcomings (Helbling 2013; Coppedge et al. 2011; Mudde and Schedler

2010). In respect to the indices we compare in this paper it is important to stress that all of

them make important and innovative contributions to the field of comparative immigration

1 By ‘index’ we understand a measurement that operationalizes in a quantitative way a social phenomenon and represents an aggregate of data. ‘Items’ and ‘indicators’ constitute the most basic elements of an index. 2 More generally, it appears that, in various fields, the construction of policy indices has become quite popular, which reflects a general tendency in political science to reduce the complexity of political phenomena and to allow for comparisons across countries and time (Pickel and Pickel 2012, Bauböck and Helbling 2011, Teorell and Lindstedt 2010, Skaaning 2010).

3

field.

es are not yet (fully) published and/or accessible we have not included them in our analy-

ses.

policy research. Critically discussing the choices researchers made when designing and com-

piling the indices is thus meant to raise awareness of how rather technical details of index

construction affect research findings substantially and to guide future research in using these

indices.3 In other words, our aim is not so much to discuss which indices are better than oth-

ers (a question that can only be answered in the light of specific research questions anyway),

as to start a more systematic methodological discussion on index building in the immigration

To the best of our knowledge, all existing indices on immigration policy output are in-

cluded in this study, as we selected the indices on the basis of an extensive literature search.4

Further indices and datasets are currently in the making which underlines how much this field

is growing (Beine et al. 2013; Helbling et al. 2013b; Peters 2013; Wong 2013). Since these

studi

To assess their methodological strengths and weaknesses, the indices will be analyzed

in detail in regard to conceptualization, measurement, and aggregation, as well as empirical

scope. Prior to this analysis, particular attention will be paid to the task of definition. “Immi-

gration policy” is a term that thus far has not been defined in a rigorous and systematic man-

ner. To assess the quality of conceptualization of the indices we therefore have to provide a

definition of immigration policy first. This definition also helps us choosing the indices we

3 By discussing immigration indices we do not intend to promote index building and large-N analyses at the expense of qualitative research. It is clear that qualitative expertise and data is necessary for building quantitative policy indices. Qualitative information allows us to better understand the values of an index, and to weight sub-parts of an index. On the other hand, we think that qualitative researchers can profit from some of the methodo-logical discussions in this paper, especially regarding the first part on conceptualization. Furthermore, qualitative researchers doing case studies might be interested in using indices to select the cases under study, e.g. by taking two countries that are at their opposites of relevant dimensions (Mill’s method of difference/similarity). 4 The following databases were searched: Worldwide political science abstracts, Sociological Abstracts, PAIS: Public affairs information service, SOWIPORT, IREON-Portal, IBSS, WISO-Sozialwissenschaften, Biblio-thekskataloge, OECD-iLibrary and Web of Science. We searched for research published in English by means of the following keywords: migration, migration policy, immigration, immigration policy, index and indicator.

4

dices have already been dis-

ussed elsewhere (Bauböck and Helbling 2011; Helbling 2013).

uilding, since “What

is badly defined is likely to be badly measured” (Nardo et al. 2005, 12).

want to evaluate: We assess indices that primarily focus on immigration policies as we define

it and exclude indices that also include immigration policy aspects but primarily set out to

measure the neighboring fields of integration and citizenship policies (e.g. Koopmans,

Michalowski, and Waibel 2012; Niessen, Huddleston, and Citron 2007). This is also justified

for reasons of parsimony: The methodological aspects of these in

c

What is immigration policy?

To assess the indices we first need to clearly define what we understand by immigration poli-

cies. This is all the more important as so far hardly any systematic definitions have been pro-

vided. This lack of rigor in turn causes the first obstacle to policy-index b

Defining a complex subject such as immigration policy is a difficult task, and disagreement

among scholars is indispensable. We thus do not argue that our definition of immigration pol-

icy should be the binding one; instead we see it as one example on how to develop a defini-

tion that is as clear and unambiguous as possible. To enable a discussion and comparison of

possible definitions it is crucial to be transparent and straightforward about the content and

the development process of the definition. Keeping this in mind, building on definitions by

Hamar (1980), Brochmann (1999), Meyers (2000), and Andreas (2003) we define immigra-

tion policy as: government’s statements of what it intends to do or not do (incl. laws, regula-

tions, decisions or orders) in regards to the selection, admission, settlement and deportation

of foreign citizens residing in the country. We arrive at this definition by addressing the fol-

lowing four questions: (1) Which groups are conceptualized as being targeted by immigration

policy? (2) Should a measure of immigration policy account for policy output, implementa-

5

enship policies? (4) How can the laws that regulate

immigration be systematically grouped?

, or have remained in the country when

their re

ferences in e.g. residence status (long vs. short term permits.) While these further segmenta-

tion and policy outcomes? (3) How can immigration policy be distinguished from the

neighboring fields of integration and citiz

Which groups are targeted by immigration policy?

Table 1 lists the groups that we define as falling under the realm of immigration policy The

first four reflect the main reasons why a state may accept immigrants (see also Helbling 2010):

labor migrants, are those individuals that states accept for economic reasons (Freeman 1978;

1979; Hollifield 1992; 1998). Members of the second group, which consists of asylum seek-

ers, recognized refugees, and people who are not recognized refugees but protected under

subsidiary/humanitarian protection are admitted for humanitarian reasons. The third group

consists of immigrants that are let in for the social reason of family reunification (Cholewin-

ski 2002; Honohan 2009). A fourth reason for immigration relates to cultural and ethnic ties,

which refers to a group that we term “co-ethnic immigrants” that also includes colonial mi-

grants (Groenendijk 2006; Jerónimo and Vink 2011). Besides these four fields there are also

policies that target a fifth group, irregular immigrants whose entry or stay is considered

unlawful. They have not been admitted for economic, humanitarian, social, cultural, or other

reasons, but have nonetheless crossed national borders

sidence permits had expired (i.e. overstayers).

It is crucial to note that individuals can move between the categories listed above, and

that further differentiations within the groups exist. E.g. within the group of labor migrants,

there are differences between high- and low-skilled labor migrants; within the group of family

reunification there are differences between how the law deals with sponsors who are citizens

and sponsors who are migrants themselves. These differences in entry status may lead to dif-

6

tions are highly useful for a variety of research questions, we found them to be too fine-

grained for the comparative purpose of this paper.

Policy outputs, policy implementation and policy outcomes

Knill and Tosun (2012, 347) state that, “policies are government statements of what it intends

to do or not to do, including laws, regulations, decisions or orders.” Easton (1965) narrows

this further by pointing to the process-like nature of policy. He defines “outputs” as “the bind-

ing decisions, their implementing actions and […] certain associated kinds of behaviour”

while outcomes are “all the consequences that flow from […] the outputs of the system”

(Easton 1965, 351). Applied to the field of immigration, legal regulations on immigration are

thus policy outputs while immigration rates, for example, are policy outcomes.5 While Easton

(1965) subsumes both legal regulations and their implementation under the heading “outputs”,

we will refer to legal regulations as policy outputs, while implementation will be regarded as

a separate aspect of policy.

Why should outcomes be accounted for in a measure of immigration policy? Money

(1999) argues that looking exclusively at the formal regulations leaves out important aspects

such as the interpretation and implementation of laws. Formal regulations do not necessarily

lead to the intended outcomes, due to diverging implementation processes, among other

things. Thus, according to her, the actual numbers of admitted immigrants would be a good

indicator for restrictiveness of immigration policies (Money 1999, 22).

Although Money has a good point in arguing for the importance of studying imple-

mentation, in this paper we will focus on immigration policy output indices. Although we

believe that measures of implementation and outcome also need to be reviewed and assessed,

we think policy output, policy implementation, and policy outcomes should be kept analyti-

5 For the immigration literature see Hollifield (1986, 114-115)

7

cally distinct for at least two reasons (Helbling 2010). First, the decision on which of the three

measures will be relevant to investigate will depend on the research question. For example, if

a researcher is interested in the effect the media has on the restrictiveness of policies, a meas-

ure focusing solely on immigration policy as formulated in law is needed. Second, it is crucial

to be able to isolate the effects of outputs on outcomes. Conflating output, implementation,

and outcome, as some indices have done in the citizenship literature (Howard 2009; Koop-

mans et al. 2005), impedes researchers from studying the independent effects policy output

might have on outcomes, and vice versa (Helbling 2013: 558-559). In the immigration policy

field, Thielemann (2003) conflates output and outcomes by including acceptance rates for

subsidiary protection in his index. This then renders it impossible to answer the question of

what effect subsidiary protection policies have on acceptance rates. However, it is important

to note that the relevant laws are sometimes not accessible, and that outcomes, such as immi-

gration rates, can serve as useful proxies for the legal situation within a country.

Demarcating immigration policies from neighboring fields

Hammar (1990) differentiates between three gates at the entrance to states in Western Europe:

entry, settlement, and full membership. This reflects the two modes of social closure of na-

tion-states, one at the territorial border (entry) and one inside the territorial border (settlement

and full membership) (see Weber 1946, 78; Brubaker 1992, ch.1; 2010). In a similar vein,

Jeanette Money proposes to study immigration control and immigrant integration as two sepa-

rate policy areas, which follow very different political logics (Givens and Luedtke 2005, 2).

Using Hammar’s classification, immigration policies are those policies that concern entry,

while integration and citizenship policies deal with settlement and full membership respec-

tively (Helbling 2013). In other words, immigration policies can be understood as “the rules

8

and procedures governing the selection, admission and deportation of foreign citizens” (Mey-

ers 2000: 1246, see also Andreas 2003, 78, and Brochmann)

However, this demarcation is not as clear-cut as it seems at first sight. Messina (2007,

23-23, citing Hammar 1985b) defines an immigrant as someone who migrates to a host coun-

try and who in theory possesses, and in practice exercises, the right to settle.6 The question is

therefore which parts of settlement regulations can be considered as immigration policy. Set-

tlement regulations stipulate what is expected from immigrants, which conditions they have to

meet to remain in a country and to become a part of the society, and the rights they are grant-

ed (Hammar 1990, 13). We believe that a measure of immigration policy also needs to incor-

porate the legal length of stay that is associated with border crossing, as well as the individual

rights that are associated with an immigrant status (see also Leblang et al. 2012; Peters 2013).

The rights that we conceptualize as belonging to immigration policies are those that enable

the immigrant to sustain a living in the host country, i.e. rights to work and rights to welfare

benefits. In other words, these are the rights that decide whether an immigrant has the material

means to continue staying in the host country. While these are arguably also part of integra-

tion policies, integration policies also cover political, social and cultural rights of immigrant

groups, i.e. they determine not only whether but also how immigrants live in the host country.

How can the laws that regulate immigration be systematically grouped?

The immigration policy of a country is comprised of a multitude of different laws. For the

sake of analytical clarity, we group these laws according to their location in the two-

dimensional scheme that is depicted in Table 1 (see also Helbling 2010): First, there is a dis-

6 According to Messina and Hammar, immigration thus means long-term settlement. In our definition we do not refer to a specific time span as it excludes those immigrants who only get temporary residence and work permits, a group of considerable size which is steadily increasing. We therefore define immigration as people moving from one nation-state to another and thereby taking up residence in the destination country. We thus exclude commuting and tourism.

9

tinction between regulations and control mechanisms (see Brochmann and Hammar 1999;

Doomernik and Jandl 2008) We call this the “modus operandi” that tells us how laws operate.

We understand regulations to be binding legal provisions that create or constrain rights (Dre-

her 2002). Controls, on the other hand, are mechanisms that monitor whether the regulations

are adhered to. To give an example: A regulation might state that immigrants need a work

permit to take up a job. The corresponding control mechanism would be sanctions for em-

ploying illegal immigrants. Controls therefore differ from implementation, because they are

formally regulated in the law.

Second, we account for the fact that states regulate and control immigration not only at

their borders, but also within their territories. We call this the “locus operandi” of laws and

thus differentiate between externally and internally targeted laws. While the external regula-

tions are comprised of conditions and eligibility criteria and the mechanisms that control

whether these criteria are abided to, the internal regulations govern the duration of stay in a

country, which rights immigrants receive and how they are monitored once they are within the

territory.

- Insert Table 1 here -

Empirical scope

We now turn to the empirical scope of the 13 indices. It is important to note that the empirical

narrowness or breadth of the indices does not tell us anything about the quality of the data,

researchers focusing on e.g. only one immigrant group or a single year are still able to con-

ceptualize, measure, and aggregate their subject matter well. However the empirical scope

restricts the range of research questions that can be addressed (Munck and Verkuilen 2002, 6).

10

Table 2 shows which policy fields targeting specific immigrant groups are covered, as well as

the temporal and spatial coverage of the indices.

As a benchmark of an all-encompassing measure of immigration policy we use the

groups listed in the definition of immigration policy given above.

- Insert Table 2 here -

As depicted in Table 2, most indices do not measure immigration policies in general but only

cover particular fields such as labor migration (Cerna 2008; Lowell 2005; Ruhs 2011) or asy-

lum (Hatton 2004; Thielemann 2003). The decision to restrict the focus is related to the re-

search questions that the authors address. To give some examples: Cerna (2008) sets out to

explain differences in high-skilled immigration policies. The explanatory variable are the par-

ties in power, labor market organization and the electoral systems are conceptualized as

intermediating factors. Ruhs (2011) analyzes how admission policies for immigrant workers

affect the rights that are granted to these workers once they are in the country, Thielemann

(2004) addresses the research question of whether increasing restrictiveness of asylum policy

has an effect on the number of asylum applications or whether other pull factors are more

decisive. Of all the indices, Klugman and Pereira (2009) and Givens and Luedtke (2005) have

the broadest empirical scope in regard to the immigrant groups they collected data on. Klug-

man and Pereira (2009) even covered the additional group of business visitors and interna-

tional students.

In regard to temporal and spatial coverage, it becomes apparent that there is a trade-off

between the span of time and the number of countries that are covered. For half of the indices,

data were collected for one to three years only whereas the other half of indices allow for the

analysis of longer periods, between twelve and seventy years. The indices that would allow

11

for longitudinal research designs are by Givens and Luedtke (2005), Mayda and Patel (2004),

Mayda (2005), Ortega and Peri (2009), Thielemann (2004), and Timmer and Williams (1998).

In regard to the number of cases included in the datasets the following picture emerges: Two

of the indices cover a relatively high number of cases; Klugman and Pereira (2009) cover 28

developed and developing countries; the Migrant Accessibility Index covers 61 countries; and

Ruhs (2011) covers 46 countries. The rest of the indices include small to medium numbers of

countries.

As of the geographical scope of the indices we observe that especially Western Euro-

pean and the traditional settler countries are covered. Only four of the studies also include

developing and middle income countries (Klugman and Pereira 2009; Economist Intelligent

Unit 2008; Oxford Analytica 2008; Ruhs 2011). Similar to the citizenship literature it thus

appears that most often Western countries with traditionally high immigration rates are ana-

lyzed where it is easier to access the data than in other countries (Helbling 2013).

As was already stated, the limited empirical scope of most of the indices prohibits re-

searchers from analyzing a number of research questions. To give two examples: When inter-

ested in whether more restrictive policies in one field of migration (e.g. labor) leads to an in-

crease of applications in another field (e.g. asylum) there is a need for longitudinal data that

differentiates between entry categories and that covers at least a medium number of countries.

One would also need to rely on data with similar characteristics to analyze the question

whether there are contagion effects between countries in regard to immigration policy’s re-

strictiveness.

A possible solution would be to merge existing datasets since they cover complemen-

tary fields. As we will see however, these indices are conceptualized in non-congruent ways.

Moreover, they often cover different periods and countries. It might therefore be difficult to

bring together these datasets. In the few cases where merging at least some parts of the data

12

would be theoretically possible (as e.g. in the cases of Cerna (2008) and Lowell (2005)), this

is prohibited by the fact that the data is not publicly available. For the same reasons it is not

possible to empirically compare the indices by e.g. running convergent validity tests7.

There are two new projects in the making, the Immigration Policies in Comparison

(IMPIC) project (Helbling et al. 2013b) and the International Migration Policy and Law (IM-

PALA) database (Beine et al. 2013), that aim at covering all aspects of immigration policies.

These new datasets will certainly allow replicating existing studies in different contexts as

well as to address new research questions and apply more sophisticated statistical methods.

Conceptualization

Conceptualization is the first and most essential step of index building, as decisions made at

this stage also have an influence on measurement and aggregation. To assess the conceptuali-

zation of the 13 indices we will look at four aspects (Munck and Verkuilen 2002). The first,

and most important of these aspects, is whether the concept to be measured is defined at all

(“definition” in Table 3). The second is whether the definition accounts for the multi-

dimensionality of immigration policy (“hierarchical structure of the concept”). The third as-

pect is whether there is any conceptual redundancy, meaning that similar aspects are included

more than once (“redundancy”). Finally, - and here we deviate from Munck and Verkuilen’s

understanding of ‘conflation’- we assess whether studies have conflated output, implementa-

tion and outcome (“conflation”).

7 Convergent validity tests try to assess in how far different measures of supposedly the same construct are corre-lated with each other. The higher the correlation coefficient the higher the likelihood that two distinct measures reflect the same construct.

13

How have existing indices defined immigration?

Have existing indices defined their subject matter, have they accounted for all aspects that we

deem important? And is their conceptualization coherent? As already argued in an earlier sec-

tion, the first and most important step for the development of a coherent and sound conceptu-

alization is 1) to give a definition and 2) to make this definition precise and unambiguous.

While three of the indices are not providing a definition at all (Mayda and Patel 2004; Ortega

and Peri 2009; Timmer and Williams 1998), and three other indices only give a very brief

definition (Economist Intelligence Unit 2008; Pham and Van forthcoming; Ruhs 2011), the

remaining six indices clearly define what they are actually aiming to measure (see Table 3).

Out of the seven indices that actually aim to measure immigration policy as a whole

(and not only a subfield such as labor migration, see Table 1), Givens and Luedtke (2005)

provide the most precise and detailed definition, while the other six are both vague and im-

precise or do not provide definitions at all. Givens and Luedtke (2005) distinguish between

immigration policy (termed “immigration control”) and integration policy (termed “immigrant

integration”), and also specify the policy areas that fall under these two broader categories

(2005, 3). Furthermore they subdivide immigration policy into four policy subfields (legal

labor immigration, family reunification, political asylum/refugees, and illegal immigration),

which are largely congruent with the immigrant groups we identified above. A shortcoming

is, however, that they do not further operationalize the policy fields into observable items, so

that in the end we do not know which specific regulations are accounted for.

While Timmer and Williamson (1998; Timmer and Williamson 1996) do not directly

define immigration policy, they make a distinction between openness and restrictiveness of

immigration policy, thereby necessarily resulting in the inclusion of different items than

would be excluded in an index solely measuring restrictiveness. Nonetheless, some more clar-

ity on why certain items are conceptualized to be attributes of immigration policies and others

14

not would have been desirable. For example, why is “selective source-country bans” (e.g.

Asians) (Timmer and Williams 1998, 741) part of the index on the anti-immigration policy

side, but “encouragement of certain source-countries” (e.g. Europeans) is not included on the

pro-immigration policy side?

Like Timmer and Williamson (1998), the Migrant Accessibility Index also neither de-

fines immigration policy nor conceptualizes policy subfields (Economist Intelligence Unit

2008). Furthermore, the indicators representing accessibility appear to have not been chosen

systematically. While “government policy towards migrants” is fairly unspecific and general,

the “power of trade union” indicator is highly specific to the labor migration policy subfield.

Moreover, of all potential criteria that could be chosen to be comprised in this subfield, the

latter neither seems to be the most important one, nor is the link to labor migration policy

immediately evident.

Good examples of precise definitions in the other indices which measure only sub-

fields of immigration policy include Oxford Analytica, which clearly restricts labor migration

programs to legal migration (2008), Lowell (2005), as well as Cerna, who restricts her index

of high skilled immigration to the science and technology sectors (2008, 11). While her defi-

nition is thus clearly stated, she herself points out that important other sectors of high-skilled

labor migration are omitted (Cerna 2008, 11). This may be considered a shortcoming when

her aim is to analyze the openness of national policies towards high-skilled immigrants in

general. Hatton (2004), considering EU policy towards asylum seekers, derives his definition

of asylum seekers from the relevant international instruments, but also does not discuss why

certain items were chosen over others.

15

Hierarchical structure of the concept

Besides defining the subject to be measured and identifying its attributes, it is also important

to specify whether the concept is one- or multidimensional and how the dimensions and re-

spective attributes are logically organized (Munck and Verkuilen 2002, 9-10). Many of the

studies also conceive of their concept as multidimensional and make a fairly similar distinc-

tion between the external and internal dimension of immigration policy to the one we de-

scribed above as our second dimension. For example, Cerna (2008) makes a distinction be-

tween admission (external) and rights (internal) of high-skilled labor migrants, and Ruhs

(2011) differentiates between the degree of openness towards labor migrants and the rights

they get once they are in the country. Likewise the authors of Oxford Analytica (2008) distin-

guish entry mechanisms from migrant workers entitlements. Klugman and Pereira (2009) dif-

ferentiate an admission and an entitlement/treatment dimension from general immigration

policy. Using different distinctions, Givens and Luedtke (2005) as well as Hatton (2004) also

perform well in specifying the hierarchical structure of the concept they are aiming to meas-

ure (see Table 3).

Nonetheless, some of the work reviewed here does not specify how the items logically

relate to each other, i.e. it remains unclear whether they are all located at the same level of

abstraction or whether some of them could be further subsumed. For example, Ortega and

Peri (2009) and Pham and Van (forthcoming) differentiate between different categories into

which laws can be grouped, but they do not elaborate on the logical relationship between the

different categories. Or, as in the case of Thieleman’s piece (2003), a hierarchical structure is

theorized in the discussion section of the article but then not implemented in the construction

of the index. Thielemann discusses three different dimensions (access control, determination

procedures and integration measures) and assigns his five measures to those three, but when

building his deterrence index, all five measures form the index with equal weight (2003, 19-

16

21). This is not transparent since he subsumed three of the five items under the integration

dimension, and thus specified a hierarchical relationship of the different items (see figure 1),

which is then not reflected in the construction of the index.

Moreover, Ortega and Peri (2009) even have a residual category of laws that could not

be classified at all. The remaining four studies do not spell out any hierarchical structure of

the concept and thus do not seem to consider immigration policy a multidimensional concept

(Economist Intelligence Unit 2008; Lowell 2005; Mayda and Patel 2004; Timmer and Wil-

liams 1998).

Redundancy

Once the attributes of a concept are identified and logically organized by level of abstraction

(see Munck and Verkuilen 2002, 8) it is important that dimensions and sub-dimensions are

mutually exclusive and similar aspects are not included in more than one category (Munck

and Verkuilen 2002, 13). This task is of course closely related to operationalizing the attrib-

utes of a concept and can thus also be understood as a part of the measurement challenge. In

regard to the reviewed indices, quite a positive picture emerges, as there are only redundancy

problems in exceptional cases.

While in general very well designed, one might wonder whether in Ruhs’ (2011, ap-

pendix) piece the item “equal pay” is sufficiently different from the item “wage restrictions”

even if they belong to two different indices. The same applies to two of Timmer and William-

son’s (1998, 741) indicators, as it remains unclear whether “subsidized or assisted passage”

can be successfully separated from “immigration treaties or contracts with shipping compa-

nies”. The Migrant Accessibility Index also faces redundancy problems, since it includes a

general indicator of “government policy towards migrants”, but also the two more specific

17

indicators, “ease of family reunification” and “ease of hiring foreign nationals”; the latter

could both be subsumed under the former (Economist Intelligence Unit 2008, appendix 2).

Conflation

Only a few of the reviewed indices have severe problems with conflation. The Migrant Ac-

cessibility Index conflates output with outcome indicators within the item “de facto and de

jure discrimination,” which makes it difficult to assess the effects of policy outputs as we

have discussed in the first part of this paper. Both Timmer and Williamson (1998) as well as

Pham and Van (forthcoming, 741) merge policy output with policy implementation by includ-

ing indicators on enforced law. The same can be criticized in Thielemann’s (2003) piece.

While the conflation of outcome and output indicators constitutes a potential pitfall for an-

swering certain research questions, it is definitely a plus that almost none of the work evalu-

ated for this paper conflates indicators that are theoretically distinguishable and may work in

opposite directions.

- Insert Table 3 here -

Measurement

After an index has been conceptualized, it needs to be measured and its components opera-

tionalized. According to Munck and Verkuilen (2002) this involves various tasks that pertain

to the selection of indicators, the selection of the measurement level and the documentation of

the coding process and thus to questions of validity and the reliability of an index (Munck and

Verkuilen 2002, 15–22). To assess the process of selecting indicators we look at which indi-

cators and sources are used (“Selection of indicators and sources” in Table 4). In regard to the

18

measurement level, we report the choice of scale (“measurement level”) and whether this

choice has been explained (“justification of measurement level”). For the documentation of

the coding process we report whether the sources where indicated in detail (“indication of

sources”) and whether the coding rules are clear and replicable (“coding rules”). Finally we

list the type of coders used (“type of coder”), whether reliability issues have been considered

(“reliability tests”), and disaggregated data provided (“disaggregated data”).

Selection of indicators and sources

According to Munck and Verkuilen (2002, 15) there are no specific rules how to choose valid

indicators. According to them it is, however, important to use various items (that can be com-

pared across cases) to account for the numerous manifestations of a concept. First and fore-

most, this concerns the number of items, which varies a lot, as we see in Table 3. Thielemann

(2003) has built the most parsimonious index, which only has five items, while the work by

Oxford Analytica (2008) and Ruhs (2011) builds on 30 and 35 items, respectively. Klugman

and Pereira (2009) even included 84 questions in their questionnaire. Simpler indices obvi-

ously require fewer resources for the data collection. In the integration and citizenship litera-

ture it has also been shown that both entire indices and sub-indices are sometimes highly cor-

related even though they have been built very differently (Helbling 2013). This might be an

argument in favor of building less complex indices. Elkins (2000) and Coppedge et al. (2011,

249-51), however, argue that more complex indices are more precise as they are more sensi-

tive to gradations and less prone to error (see also Bader 2007, 876).

Whether a low or high number of indicators are required for the operationalization de-

pends, of course, on how a social phenomenon is conceptualized. Assessing the selection of

indicators is rather difficult in the immigration policy field where, as we have seen above, the

19

concepts under study have often not been defined precisely. Even in the cases where the con-

cept is well defined, a detailed discussion of why specific items have been chosen is often not

provided. It is therefore difficult to tell whether the indicators measure what they are supposed

to measure. Notable exceptions are Thielemann (2003, 18-21) and Ruhs (2011, 8-21), who

discuss their individual indicators in detail. One might of course still wonder why certain

items are included and other excluded. In their cases it is however at least possible to follow

their reasoning.

To guarantee a certain degree of validity, Munck and Verkuilen (2002, 15–7) also pro-

pose to rely on multiple as well as independent sources and thereby to pay attention to any

systematic source of measurement error. A source might be judged invalid if it is based on

subjective assessments (for example when experts are invited to give their own opinion) or

biased (for example when state sources are used that present their country in a particular

light).

In the field of immigration policy literature invalid sources are not that much of a

problem as the indices measure concrete formal rules. As we see in Table 4, with the excep-

tion of the Migrant Accessibility Index all indices rely mostly on immigration laws them-

selves. In many cases, the data has been collected across a variety of sources. Unfortunately,

often it does not become clear to what extent the sources have been used to cross-validate the

data or simply as supplements if certain sources did not provide all the information needed.

As long as legal texts are accessible it is rather straightforward to assess whether a cer-

tain rule exists or not. A legal text in-itself cannot be biased as it is the very element that is

measured. The question is rather whether all relevant legal texts have been assessed. Not only

might immigration issues be regulated in primary and secondary laws as well as administra-

tive guidelines. It might also be the case that, for example, labor market or social security leg-

islation deals with immigration issues. It is thus not always straightforward to judge whether

20

all relevant documents have been analyzed and where to draw the line between policy output

and implementation.

Givens and Luedtke (2005, 11) as well as Cerna (2008, 12) explicitly choose a very

broad understanding of what a law is and not only analyze acts of parliament but also execu-

tive regulations, decrees, circulars and administrative rulings. Pham and Van (forthcoming, 7)

exclude laws that have no practical effect on immigrants and resolutions that are not imple-

mented. Some of the other reviewed indices, however, do not provide such information, and it

is therefore not possible to replicate the measurement.

This lack of information becomes even more problematic if the data has not been col-

lected by the researchers themselves but has been provided by experts, as it is the case for

some indices (Oxford Analytica 2008; Migrant Accessibility Index; Klugman and Pereira

2009). None of the studies indicates what instructions were given to the experts. We thus do

not know whether the experts answered the questions from memory, consulted secondary lit-

erature, or relied on all relevant legal texts. Another problem concerns the fact that with the

exception of Givens and Luedtke (2005) all studies indicate to have collected their data on

immigration laws mostly through websites, reports and secondary literature. However, they

have not indicated how the data has been obtained through these secondary sources. It is thus

impossible to assess their validity. It might very well be that all relevant regulations were pro-

vided in their original formulations (for example on the websites of national authorities). It

might however also be that for example international reports have already made an initial se-

lection of laws and summarized the information.

21

Measurement level and justification of measurement level

A valid index also needs to be able to capture the variety of empirical cases without however

providing a measurement that is unnecessarily detailed (Munck and Verkuilen 2002, 17-8).

Since it is quite hard to decide on the appropriate measurement level Munck and Verkuilen

(2002) propose to assess whether a study at least discusses and justifies the measurement level

of its indicators and thus the questions of how many values an indicator can take or whether

for example an indicator constitutes an ordinal or a ratio scale. None of the indices however

provides detailed justification why a certain level has been chosen. Oxford Analytica (2008,

16) briefly discusses the problem, however without giving the reader a clear idea how they

solved this problem. Most indices allow for three to eleven steps to differentiate between

more or less restrictive/liberal policies. It is however not specified why more or less steps on a

scale are chosen.

Ortega and Peri (2009), Mayda (2005) and Hatton (2004), who are primarily interested

in policy changes, only allow for two to three values indicating whether a policy has become

more restrictive, more liberal, or did not change. In a similar vein, Pham and Van (forthcom-

ing) differentiate between laws that are restrictive or benefit immigrants. Neither of these four

projects, nor Timmer and Williams (1998), operationalize immigration laws by means of in-

dividual, clearly defined items. Instead they assign one score to all laws in force at a given

point in time.

Whether the measurement levels are appropriate depends on how policies have been

conceptualized (and as we have seen above, we know little of how some of the existing indi-

ces are conceptualized) and especially whether one thinks that national policies can be

grouped into different types or whether they can be put on a continuous index that varies be-

tween very restrictive and very generous.

22

Sources and coding rules

As of the indication of sources we see that, besides Lowell (2005) and Klugman and Pereira

(2009), all the indices indicate their sources. However, they often only provide very general

information that makes it difficult to find the respective documents.

All the indices provide information on the coding rules and thus make it possible to

replicate their codings, at least to a certain extent. Thielemann (2004) as well as Ruhs (2011)

constitute particularly good examples of how to provide coding procedures: Their indices

consist of a series of dichotomous items that measure whether a certain regulation exists or

not. We also consider the index by Mayda (2005) as clear and replicable as it simply indicates

whether there was change at all (under the assumption that we know whether a new law

means more restrictiveness or generosity).

On the other hand, Timmer and Williams (1998), Hatton (2004), Lowell (2005), and

Givens and Luedtke (2005) make it quite difficult to replicate their findings. Most often vague

terminology is the problem: How would one assess “modest” advertising of immigration pol-

icy (Timmer and Williams 1998, 741) or a “significant” increase in restrictiveness (Hatton

2004, 30)? Givens and Luedtke (2005, 10) provide examples of specific laws to indicate what

they understand by a highly restrictive or expansive law. But even if one is familiar with these

examples, it does not become clear why exactly these laws can be considered as particularly

restrictive or expansive.

In between, there are a series of studies whose coding rules are mostly clear, even if

they are partly based on subjective evaluations (Cerna 2008; Klugman and Pereira 2009; Or-

tega and Peri 2009; Oxford Analytica 2008). In their works, there are a small number of indi-

cators that are based on vague coding instructions, which can only be responded to with a

certain degree of subjectivity. For example, Klugman and Pereira (2009, 25) ask experts how

23

difficult it is to renew/extend a temporary visa and provide rather vague answer categories

such as “very difficult”, “somewhat difficult” etc.

Coders, reliability and disaggregated data

Information on the persons who coded the data is extremely parsimonious. Only Givens and

Luedtke (2005), Ortega and Peri (2009), Ruhs (2011), and the Migrant Accessibility Index

(Oxford Analytica 2008) provide some information. Thielemann (2003) indicated that he did

the coding himself. It can be assumed that in the remaining studies it was also the authors who

did the coding. If only one person did the coding this raises serious problems as this might

lead to significant bias especially if the coding rules are not clearly specified (Munck and

Verkuilen 2002, 19).

The poor information on coders is also reflected in the fact that reliability is one of the

most neglected aspects for all immigration policy indices. The exceptions are Givens and

Luedtke (2005, 10), Ruhs (2011, 27), Klugman and Pereira (2009, 5) and Ortega and Peri

(2009, 8), who at least discuss reliability issues and indicate that their data has been cross-

checked to a certain extent. It however remains unclear how often different sources and cod-

ers have led to conflicting answers or how exactly this problem has been solved in their stud-

ies. All other indices do not provide any information about the reliability of their data.

Reliability might be less of an issue in a field that bases its indices on concrete formal

regulations that are rather easy to interpret (especially if the indices consist of clearly speci-

fied dichotomous items). It becomes, however, more of a problem if the specific legal sources

on which the assessments are based are unclear. It might always be the case that coders under-

stand the coding instructions differently. Especially, if they are asked to capture the “political

sentiment” of immigration regulations, this often meant “judgment calls“ as is the case in the

24

study by Timmer and Williamson (1996, xiii). Moreover, the legal texts might still be inter-

preted to a certain extent, especially in the light of the fact that laws are formulated differently

across countries and that the legislators sometimes leave considerable interpretative room.

Thus, while it might be easy to answer a certain question in one country, the same issue might

be regulated in a different way in another country, which in turn might make it more difficult

to answer the question—this poses a serious challenge to cross-country comparisons.

The final measurement issue concerns the question whether the data is also accessible

in a disaggregated form, which would make it possible to better understand how an index has

been built. Some of the indices look at disaggregated data and investigate subindices. They

however hardly ever provide the data that would allow others to replicate their findings and to

assess the appropriateness of their measurement levels. This is also true for the datasets them-

selves. Contrary to, for example, the integration and citizenship literature, where all datasets

are publicly available (Helbling 2013, appendix), in the immigration literature, only a very

few studies make their data (partly) available (see Table 3).

- Insert Table 4 here -

Aggregation and weighting

Coppedge et al. (2011, 250) emphasize that all multi-faceted indices must struggle with the

challenge of aggregation, including decisions on which items to combine into an index,

whether to add or multiply them and how to weight the different items. However, this does

not mean that all indices necessarily must consist of one or several aggregated scores. An in-

dex can also be comprised of several numerical expressions representing the multiple dimen-

sions of the construct at hand. Mayda and Patel (2004), Mayda (2005), Ortega and Peri

25

(2009), Givens and Luedtke (2005), Pham and Van (forthcoming) as well as Timmer and Wil-

liams (1998) all applied the same type of operationalization and scoring technique where each

unit of analysis i.e. the year or law is given one overall score, hence the five indices are al-

ready at the highest aggregate level, i.e. the level of restrictiveness of immigration policy as a

whole, and the challenge of aggregation is not relevant.8

While not aggregating and analyzing items independently is a choice too, all analysts

building an index should make their aggregation choice explicit. Moreover, one must keep in

mind that by moving to a higher level of aggregation, information might be lost, which can

lead to a loss of validity (Munck and Verkuilen 2002, 22). According to Munck and Verkuilen

(2002) the choice of aggregation level is therefore the first task analysts must undertake in

regard to aggregation. The second task is the selection of aggregation rules and the third task

is documentation of aggregation rules and the aggregated data (Munck and Verkuilen 2002,

Traunmüller 2012, 15) (see also Table 5).

Aggregation level and justification of aggregation level

When choosing the level of aggregation, analysts must “balance the desire for parsimony and

the concern with underlying dimensionality and differentiation” (Munck and Verkuilen 2002,

22). When looking at the eight indices, all of which face the challenge of choosing an aggre-

gation level, the desire for parsimony seems to have overruled any other concern. The general

trend among the indices is to proceed to the highest level of aggregation without much “bal-

8 Although they are at the highest aggregate level and the challenges of aggregation do not apply, both Givens and Luedtke (2005), Pham and Van (forthcoming) and Timmer and Williamson (1998) address the problem of 1) how to score when several laws are in force within the same year and 2) the problem of scoring laws that are restrictive towards certain immigrants and open towards others. Givens and Luedtke (2005) average the scores in order to construct one overall score for each year. Pham and Van (forthcoming, 7-8) assign scores, weight the scores based on the expected impact of the laws and then add the weighted scores. Timmer and Williamson (1998, 742) “add up the positive and negative attributes” when scoring. There is no information on how Mayda (2005), Mayda and Patel (2004) or Ortega and Peri (2009) addressed these problems.

26

ancing” of multi-dimensionality. All of the indices aggregate at the highest level. In addition,

four of the indices include aggregated scores at the sublevels, producing one overall score and

a number of subscores (Cerna 2008; Hatton 2004; Oxford Analytica 2008; Ruhs 2011), and

Klugman and Pereira (2009) include some of their individual items in their analyses (see Ta-

ble 4). As the appropriate level of aggregation is based on the above mentioned balancing, it

is difficult to evaluate the level of aggregation, hence we propose to assess whether a study

discusses and justifies the level of aggregation. None of the indices does this. The appropri-

ateness of reducing data into single score(s) is not theoretically justified or tested, which

weakens the validity of the resulting scores (Munck and Verkuilen 2002, 23). In addition, the

hierarchical structure of the index might be lost in the process. An example is Oxford Ana-

lytica’s Labour Migration Policy Index. Although it is clear and specific in implementing a

hierarchical structure, this very structure is disregarded as aggregation only takes place at the

highest and second highest levels, with all items at the lower levels treated as being on the

same level despite the fact that they are theorized differently (Oxford Analytica 2008).

Selection of aggregation rule and weighting scheme

When it comes to actually weighting and aggregating there is no gold standard. Aggregation

rules for democracy indices are typically additive, with an implicit or explicit weighting

scheme (Coppedge et al. 2011, 250). The same is the case for immigration policy indices. In

contrast to the other indices, The Migrant Accessibility Index (Economist Intelligence Unit

2008) averages scores instead of adding them. The overall score of the Labour Migration Pol-

icy Index is also averaged, whereas the subscores are calculated by adding the item scores

(Oxford Analytica 2008). The aggregation rule used by Hatton (2004) and Klugman and Pe-

reira (2009) is unclear, but most likely additive and the aggregation rules of the remaining

27

).

five indices are all additive. Out of these five, two include an explicit weighting scheme:9

Oxford Analytica (2008) and Ruhs (2011) (Table 5

It is important to note that not applying weights is not necessarily the same as not ap-

plying a weighting scheme. If the number of items per component differs, not applying

weights can easily lead to giving more weight to the specific component of the index. If the

number of items per sub-field or sub-dimension differs, and the item scores are added, not

applying weights would result in giving more weight to the subfields with the most items. On

the other hand, averaging across items without applying weights would result in giving more

weight to the items, making up the subfield with the lowest number of items.

Different means of aggregation might lead to very different solutions (Coppedge et al.

2011, 250), hence the process of weighting and aggregating should always be driven by the-

ory. According to Munck and Verkuilen (2002, 26), a two-step process should be used when

selecting the aggregation rule: First, the analyst “must make explicit the theory concerning the

relationship between attributes” and secondly, he or she “must make sure that there is a corre-

spondence between the theory and the selected aggregation rule”. To assess this task we there-

fore analyze if the aggregation rule and weighting scheme is theoretically justified, whether

the chosen scheme corresponds with the applied aggregation rule and weighting scheme and if

the aggregation rule has been tested.

Three of the indices, Ruhs’ (2011) indices on “openness” and “legal rights,” Oxford

Analytica’s (2008) Labour Migration Policy Index and the Migrant Accessibility Index

(Economist Intelligence Unit 2008) provide a justification of their aggregation rule. The re-

maining five indices do not adequately provide a theory based justification of the aggregation

rule. Thielemann (2003, 21) comes close with a statement on his five policy measures having

9 “Applying a weighting scheme” means that the weight of each item is explicitly accounted for in the aggrega-tion rule and/or that the scores are weighted.

28

the same potential deterrence effect, but he does not provide theoretical support for this state-

ment.

Ruhs (2001) justifies his aggregation procedure based on the idea of equal weights

with the argument that there is no objective way of assessing whether some items matter more

than others. Hence, transparency and simplicity is preferred, and he adds normalized, un-

weighted scores. As mentioned above, the idea of equal weights (in the sense of not applying

weights) when adding the scores rely on the number of items per sub-dimension being equal,

which is not the case in Ruhs’ Indicator of Migrant Rights where one sub-dimension (family

reunion for migrant workers and employment rights of family members/dependents) com-

prises fewer items than the rest (Ruhs 2011, appendix 3). In this context it is worth quoting

Decancq and Lugo who stress that equal weighing is a type of weighting scheme too, and that

taking decisions regarding the relationship of the items is unavoidable: ”there is no escape

from the fact that weights reflect an important aspect of the trade-offs between the dimensions

so that agnosticism cannot be achieved” (2013, 23).

The consideration of the relationship between the items (with the exception of above

mentioned omission of hierarchical structure) is well reflected in the weighting scheme of the

Labour Migration Policy Index. The authors argue that prominence should be given to pro-

grams (items) that result in high inflow of migrant workers (Oxford Analytica 2008, 18).

They start by adding and normalizing items scores and then they apply weights based on the

latest available number of annual inflows of migrants under a particular migration program.

Besides conflating immigration output and outcome, this approach results in a measure that

varies with changes in immigration patterns, hence comparisons across time and across coun-

tries are not straightforward. Thorough considerations of the relationship between the items

are also found in the Migrant Accessibility Index, who convert the scores to 0-100 and calcu-

late a weighted average. The authors apply a third approach and apply weights based on sev-

29

eral external experts’ evaluation (Economist Intelligence Unit 2008, 6). By consulting several

experts a more systematic representation of the diversity of judgments may be achieved as the

arbitrariness of following the opinion of only one expert is avoided (Decancq and Lugo 2013,

24). The number of experts, their professional knowledge and the technique applied is not

specified, though, and bias due to the selection of experts might occur. In addition, the ex-

perts’ judgment might not be consistent with the “real world”.

The remaining three indices do not make any explicit assumptions on the relationship

between the measured items (Cerna 2008; Lowell 2005; Thielemann 2003). In addition, none

of the proposed aggregation rules are tested, and the robustness of all the immigration policy

indices is questionable (Table 5).

Documentation aggregation rules and the aggregated data

Munck and Verkuilen (2002, 25) stress that aggregation rules and aggregated data should be

publicized to allow for aggregation tests and replications of aggregation procedures. Five out

of the eight indices do not provide any documentation on the aggregation rules or the aggre-

gated data (see Table 5). The Migrant Accessibility Index publicizes the weights of each item

but not the aggregated data necessary for replication and tests (Economist Intelligence Unit

2008). Oxford Analytica (2008) provides detailed information, however, without providing

the weights. This makes it difficult to follow the last step of the aggregation. Ruhs (2011)

provides the aggregated data per country together with some descriptive statistics, but unfor-

tunately only for one of his two indices. This makes it difficult for future indices to use the

existing indices for means of validity tests.

- Insert Table 5 here -

30

Conclusion

Doing comparative research on immigration is a challenging but compelling task that requires

valid, reliable, and replicable measures. In that regard over the last decade we have observed

an important development in the immigration literature: A great many indices have been con-

structed that allow us to tackle a series of research questions in a more systematic way.

The aim of this paper was to give an overview of the work that has been done so far, to

present the strengths of the existing indices, but also to point out to several methodological

weaknesses that need to be addressed in the next phase of immigration policy research. To

assess the quality of the indices, the slightly adapted framework developed by Munck and

Verkuilen (2002) for democracy measures turned out to be extremely useful, as it allowed us

to compare in detail a large variety of differently built indices.

In regard to conceptualization it became apparent that “immigration policy” is often

not defined, as if it was clear what was understood by this term. When assessing the indicators

used in the respective indices it however turned out that the “immigration policy” that one

index represents is compiled using very different indicators than another index; various re-

searchers thus have different understandings of what immigration policies consist of. Our own

definition made clear that the various steps of the policy cycle, demarcation from neighboring

policy fields and the distinction of various kinds of regulations within the immigration policy

field need to be discussed to make clear what we understand by this policy field. This is the

first major lesson to be learned from our overview: “Immigration policy” is a more complex

social phenomenon than one might think. It needs to be defined, not only to make clear what

we talk about but also to be able to assess how the respective indices are measured and aggre-

gated.

31

In that regard, we saw that the lack of thorough methodological discussions sometimes

leads to indices that seem to be measured and aggregated without following theoretically

grounded rules. It needs, of course, to be emphasized that there are no general rules of index

building—and there do not need to be such rules—since researchers should build their indices

tailored to the research questions they are interested in answering. However, in this process it

is of utmost importance to think about conceptualization, measurement, and aggregation. And

this is the second major lesson we learned: While different methodological choices are often

possible it is crucial to discuss these choices in a transparent way so that other researchers

understand how an index has been built.

With this being said, the indices assessed in this paper generally perform very well in

regards to one or more of the steps of the index-building process and thereby provide impor-

tant ideas for future index building. Cerna (2008), Givens and Luedtke (2005), and Hatton

(2004) and Lowell (2005) all clearly define their construct at hand without imposing redun-

dancy or conflating policy output and outcome. Givens and Luedtke (2005) furthermore pro-

vide a sound description of their measurement. The same is the case for, Oxford Analytica

(2008), Ruhs (2011), Thielemann (2003), Ortega and Peri (2009), and the authors of the Mi-

grant Accessibility Index (Economist Intelligence Unit 2008). Overall, all of them perform

well in regards to how they conceptualize their constructs although facing problems in regards

to either defining the subject at hand, specifying its hierarchical structure, conceptual redun-

dancy or conflation of outputs with implementation and outcome. In addition, Oxford Ana-

lytica (2008), the authors of the Migrant Accessibility Index (Economist Intelligence Unit

2008), and Ruhs (2011) and Thielemann (2003) all provide great insight into their aggregation

and weighting procedures.

Taken together, all of these methodological considerations should allow us to improve

future index building and to make the use of policy indices even more attractive. One of the

32

biggest advantages of an index is its parsimony. Instead of having to look at multiple vari-

ables, an index provides a numeric value representing for example how restrictive a country is

towards immigration, thereby condensing large amounts of information into an easy to use

single measure. This parsimony however can be counterproductive and obtrusive to producing

insight and knowledge, particularly if the users of those indices are not aware of the methodo-

logical choices taken while building the index. For example, we proposed that immigration

policy can be subdivided into five subfields of immigrant groups (economic, humanitarian,

family, co-ethnics, and irregular immigrants), but whether all four are equally important de-

pends on the theoretical (normative or positivist) framework. A country that allows co-ethnic

immigration only may still be ranked rather open under a weighting and aggregation scheme

where all dimensions count equally, but rather restrictive under a scheme where more weight

is put on economic and humanitarian considerations. Thus, users should be aware of the theo-

retical framework and expectations guiding the index building and where necessary adjust

those schemes to fit their own theory and research question and not blindly ‘plug in’ a single

number measure because it is convenient and easy to do so.

33

List of references

Andreas, Peter. 2003. "Redrawing the Line: Borders and Security in the Twenty-First Cen-

tury." International Security 28 (2):78–111.

Bader, Veit. 2007. "The Governance of Islam in Europe: The Perils of Modelling." Journal of

Ethnic and Migration Studies 33(6): 871-886.

Bauböck, Rainer, and Marc Helbling, eds. 2011. “Which indicators are most useful for com-

paring citizenship policies?” Fiesole: RSCAS Working Paper 2011/54.

Beine, Michel, Brian Burgoon, Mary Crock, Justin Gest, Michael J. Hiscox, Patrick McGov-

ern, Hillel Rapoport and Eiko Thielemann. 2013 “Dilemmas in Measuring Immigration

Policies and the IMPALA Database.” Migration and Citizenship. Newsletter of the Ameri-

can Political Science Association. Organized Section on Migration and Citizenship 1(2):

15-22.

Brochmann, Grete and Tomas Hammar (eds.). 1999. Mechanisms of Immigration Control. A

Comparative Analysis of European Regulation Policies. Oxford and New York: Berg.

Brochmann, Grete (1999). “The Mechanisms of Control”, Grete Brochmann and Tomas

Hammar (eds.) (1999). Mechanisms of Immigration Control. A Comparative Analysis of

European Regulation Policies. Oxford and New York: Berg, 1-27.

Brubaker, Rogers. 1992. Citizenship and Nationhood in France and Germany. Cambridge,

MA: Harvard University Press.

Cerna, Lucie. 2008. “Towards an EU Blue Card? The delegation of National High Skilled

Immigration Policies to the EU level.” COMPAS Working Paper No. 65. Oxford: ESRC

Centre on Migration, Policy and Society. Oxford.

34

Cholewinski, Ryszard. 2002. "Family Reunification and Conditions Placed on Family Mem-

bers: Dismantling a Fundamental Human Right." European Journal of Migration and Law

(4):271–90.

Coppedge, Michael, John Gerring, David Altman, Michael Bernhard, Steven Fish, Allen

Hicken, Matthew Kroenig, Staffan I. Lindberg, Kelly McMann, Pamela Paxton, Ernest

Sergenti, Svend-Erik Skaaning, Jeffrey Staton, and Jan Teorell. 2011. "Conceptualizing

and Measuring Democracy: A New Approach." Perspectives on Politics 9 (02):247–67.

Decancq, Koen and María Ana Lugo. 2013. "Weights in multidimensional indices of wellbe-

ing. An overview." Econometrics Reviews 32 (1):7–34.

Doomernik, Jeroen and Michael Jandl (eds.). 2008. Modes of Migration Regulation and Con-

trol in Europe. Amsterdam: Amsterdam University Press.

Dreher, Sabine. 2002. "Regulation." In Lexikon der Politikwissenschaft, eds. Dieter Nohlen,

and Schultze Rainer-Olaf. 2 vols. München: C.H.Beck, 804–5.

Easton, David. 1965. A Systems Analysis of Political Life. Chicago: University of Chicago

Press.

Economist Intelligence Unit. 2008. Global Migration Barometer. Methodology, Results and

Findings, London: Western Union, The Economist.

Elkins, Zachary. 2000. "Gradations of Democracy? Empirical Tests of Alternative Conceptu-

alizations." American Journal of Political Science 44 (2):293–300.

Freeman, Gary P. 1979. Immigrant Labor and Racial Conflict in Industrial Societies: The

French and British Experience. Princeton: Princeton University Press.

–––. 1978. "Immigrant Labor and Working-Class Politics: The French and British Experi-

ence." Comparative Politics 11 (1):25–41.

35

Givens, Terri and Adam Luedtke. 2005. "European Immigration Policies in Comparative Per-

spective: Issue Salience, Partisanship and Immigrant Rights." Comparative European Poli-

tics 3 (1):1–22.

Groenendijk, Kees. 2006. "The status of quasi-citizenship in EU member states: Why some

states have 'almost citizens'." In Acquisition and Loss of Nationality. Policies and Trends

in 15 European States, eds. Rainer Bauböck, Eva Ersbøll, Kees Groenendijk, and Harald

Waldrauch. Amsterdam: Amsterdam University Press, 411–29.

Hatton, Timothy. 2004. "Seeking asylum in Europe." Economic Policy 19 (38):5–62.

Hammar, Tomas. 1990. Democracy and the Nation State. Aliens, Denizens and Citizens in a

World of International Migration. Aldershot: Avbury.

–––. 1985. “Immigration Regulation and Alien Control”. In: Tomas Hammar (ed.). European

Immigration Policy: A Comparative Study. Cambridge: Cambridge University Press.

Helbling, Marc 2010. Immigration Policies in the Western World: New Indicators, Causes

and Effects. Project Proposal German Research Foundation.

–––. 2013. "Validating Integration and Citizenship Policy Indices." Comparative European

Politics 11(5): 555-576.

Helbling, Marc, Liv Bjerre, Friederike Römer and Malisa Zobel, eds. 2013a. “How to meas-

ure immigration policies.” Migration and Citizenship. Newsletter of the American Political

Science Association. Organized Section on Migration and Citizenship 1(2): 4-8.

Helbling, Marc, Liv Bjerre, Friederike Römer and Malisa Zobel, eds. 2013b. “The Immigra-

tion Policies in Comparison (IMPIC) Index: The Importance of a Sound Conceptualiza-

tion.” Migration and Citizenship. Newsletter of the American Political Science Associa-

tion. Organized Section on Migration and Citizenship 1(2): 8-15.

36

Hollifield, James F. 1998. "Migration, Trade and the Nation-State: The Myth of Globaliza-

tion." UCLA Journal of International Law and Foreign Affairs 3 (2):595–636.

–––. 1992. Immigrants, Markets, and States. The Political Economy of Postwar Europe.

Cambridge: Cambridge University Press.

–––. 1986. “Immigration Policy in France and Germany: Outputs versus Outcomes”, Annals

of the American Academy of Political and Social Science 485: 113-128.

Honohan, Iseult. 2009. "Reconsidering the Claim to Family Reunification in Migration." Po-

litical Studies 57 (4):768–87.

Howard, Marc Morjé. 2009. The Politics of Citizenship in Europe. New York: Cambridge

University Press.

Jerónimo, Patricia and Maarten Peter Vink. 2011. "Citizenship in a Postcolonial Context:

Comparing Portugal and the Netherlands." Portuguese Journal of Political Science and In-

ternational Relations 6:109–29.

Klugman, Jeni and Isabel Medalho Pereira. 2009. Assessment of National Migration Policies:

An emerging picture on admissions, treatment and enforcement in developing and devel-

oped countries. HDRP-2009-48. Human Development Report Office (HDRO), United Na-

tions Development Programme (UNDP).

Knill, Christoph and Jale Tosun. 2012. Public Policy: A new Introduction. Basingstoke: Pal-

grave Macmillan.

Koopmans, Ruud, Ines Michalowski, and Stine Waibel. 2012. "Citizenship Rights for Immi-

grants: National Political Processes and Cross-National Convergence in Western Europe,

1980–2008 1." American Journal of Sociology 117 (4):1202–45.

37

Koopmans, Ruud, Paul Statham, Marco Giugni, and Florence Passy, eds. 2005. Contested

Citizenship. Immigration and Cultural Diversity in Europe. Minneapolis: University of

Minnesota Press.

Leblang, David A., Jennifer Fitzgerald, and Jessica Teets. 2012. “The Political Economy of

International Migration.” University of Virginia. Working Paper.

Lowell, Lindsay B. 2005. Policies and Regulations for Managing Skilled International Mi-

gration for Work. New York: United Nations Expert Group Meeting on International Mi-

gration and Development.

Mayda, Anna Maria. 2005. "International Migration: A Panel Data Analysis of Economic and

Non-Economic Determinants." IZA Discussion Papers No. 1590. Bonn: Institute for the

Study of Labor.

Mayda, Anna Maria, and Patel, Krishna. 2004. OECD Countries Migration Policy Changes.

Messina, Anthony M. 2007. The Logics and Politics of Post-WWII Migration to Western Eu-

rope. New York: Cambridge University Press.

Meyers, Eytan (2000). “Theories of International Immigration Policy – A Comparative Anal-

ysis”, International Migration Review 34(4): 1245-1282.

Money, Jeannette. 1999. Fences and neighbors. The political geography of immigration con-

trol. Ithaca, N.Y: Cornell University Press.

Mudde, Cas and Andreas Schedler. 2010. "Introduction: Rational Data Choice." Political Re-

search Quarterly 63 (2):410–6.

Müller, Thomas and Susanne Pickel. 2007. "Wie lässt sich Demokratie am besten messen?

Zur Konzeptqualität von Demokratie-Indizes." Politische Vierteljahresschrift 48 (3):511–

39.

38

Munck, Gerardo L. and Jay Verkuilen. 2002. "Conceptualizing and Measuring Democracy:

Evaluating Alternative Indices." Comparative Political Studies 35 (1):5–34.

Nardo, Michael, Michaela Saisana, Andrea Saltelli, Stefano Tarantola, Anders Hoffman, and

Enrico Giovannini. 2005. "Handbook on Constructing Composite Indicators: Methodology

and User Guide." OECD.

Niessen, Jan, Thomas Huddleston, and Laura Citron. 2007. Migrant Integration Policy Index

(MIPEX). British Council and Migration Policy Group.

Ortega, Francesco and Giovanni Peri. 2009. "The Causes and Effects of International Labor

Mobility: Evidence from OECD Countries 1980-2005." Human Development Research

Paper (6):1–42 (Accessed January 02, 2012).

Oxford Analytica. 2008. Labour Migration Policy Index Phase II. IOM, October 01, 2008.

Pham, Huyen and Pham Van. Forthcoming. "Measuring the Climate for Immigrants: A State-

by-State Analysis." In The Role of the States in Immigration Enforcement and Policy, eds.

Gabriel Jack Chin, and Carrissa Hessick. New York: New York University Press.

Peters, Margaret. 2013. “Open Trade, Closed Borders: Immigration in the Era of Globaliza-

tion.” Yale University. Working Paper.

Pickel, Susanne and Gert Pickel. 2012. "Die Messung von Indizes in der Vergleichenden Poli-

tikwissenschaft – methodologische Spitzfindigkeit oder substantielle Notwendigkeit." Zeit-

schrift für Vergleichende Politikwissenschaft 6(1): 1-17.

Ruhs, Martin. 2011. “Openness, Skills and Rights: An Empirical Analysis of Labour Immi-

gration Programmes in 46 High- and Middle-Income Countries.” COMPAS Working Paper

No.88. Oxford: ESRC Centre on Migration, Policy and Society.

Skaaning, Sven-Erik. 2010. "Measuring the Rule of Law." Political Research Quarterly

63(2): 449-460.

39

Teorell, Jan and Catharina Lindstedt. 2010. "Measuring Electoral Systems." Political Re-

search Quarterly 63(2): 434-448.

Thielemann, Eiko R. 2004. "Why Asylum Policy Harmonisation Undermines Refugee Bur-

den-Sharing." European Journal of Migration and Law 6:47–65.

–––. 2003. “Does Policy Matter? On Governments' Attempts to Control Unwanted Migra-

tion.” IIIS Discussion Papers No.9, Dublin: Institute for International Integration Studies,

Trinity College.

Timmer, Ashley S. and Jeffrey G. Williams. 1998. "Immigration Policy Prior to the

1930s: Labor Markets, Policy Interactins and Globalization Backlash." Population and

Development Review 24 (4):739–71.

Timmer, Ashley S. and Jeffrey G. Williamson. 1996. "Racism, Xenophobia or markets. The

Political Economy of immigration policy prior to the thirties." NBER Working Paper Se-

ries (5867).

Traunmüller, Richard. 2012. "Zur Messung von Staat-Kirche-Beziehungen: Eine vergleichen-

de Analyse neuerer Indizes." Zeitschrift für Vergleichende Politikwissenschaft 6 (1):207–

31.

Weber, Max. 1946. From Max Weber: Essays in Sociology. New York: Oxford University

Press.

Wong, Tom K. 2013. “Rights, Deportation, and Detention in the Age of Immigration Con-

trol.” University of California, San Diego. Working Paper.

Tables and figures

Table 1: A conceptualization of immigration policy

Policy dimensions Policy fields

Modus Operandi

Locus Operandi

Labor migration

Asylum / refugees

Family reunification

Co-ethnics Irregular

Immigrants

External Regulation

Internal

External Control

Internal

Table 2: Empirical scope

Name of Index Immigrant groups covered Temporal coverage Spatial coverage Availability of dataCerna 2008 Labor, high skilled immigration 2007 (p. 11) 20 (Same as Low ell 2005 plus

additional countries) (p.11)Not publically available.

Givens/Luedtke 2005

Immigration policy (asylum; family reunifcation; labor; irregular immigration; citizenship; anti-discrimination)

1990-2002 (p.1) 3 countries (DE; FR; GB) (p.1) Not publically available.

Hatton 2004 Asylum 1981-1999 (p. 8) EU 15 (except Luxembourg) (p. 8) Not publicly available

Klugman/Pereira 2009

Labor migration, family reunif ication, asylum/refugees, international visitors (business), international students.

2009 (Abstract) 28 developed and developing countries (p. 3)

Not publicly available

Lowell 2005 Labor, high skilled immigration 2001 (p. 11) 12 (Main European receiving countries, traditional countries of immigration, South Africa and Japan) (p i)

Not publicly available

Mayda 2005; Mayda & Patel 2004

probably: Labor, asylum, family, control for analysis: 1980-1995 (but differs from coutnry to country) (e.g. Mayda&Patel 2004: 2 for Australia 1948-2002)

14 OECD countries + EU: US, UK, CH, SE, NO, NL, LU, JP, DE, FR, DK, CA, BE, AU, EU (Mayda 2005: 1)

Appendix on changes, but scored and aggregated data not available

Migrant Accessibility Index (part of the Global Migration Barometer 2008)

(accessibility for) migrants 2007 (p. 24) 61 countries (p. 17) Not publicly available

Ortega & Peri 2009 Entry of non-asylum immigrnats, Entry and/or stay of non-asylum immigrants, entry and/or stay of asylum seekers

1980-2005 (p. 1) 14 OECD countries (AU, BE, CA, DK, FR, DE, JP, LU, NL, NO, SE, CH, UK, US) (p.1)

Not publicly available

Oxford Analytica 2008

Labor migration 2005-2007 (not all three years covered individually but the law s that date from around this time) (p. 5)

13 countries (AE, AU, CA, DE,IN, IT, JP, NZ, NO, ES, SG, UK, US) (p. 5)

Data available in appendix.

Pham & Van 2012 Law enforcement, employment, language, housing, benefits, voting, legal benefits

2005-2009 50 US states Not publicly available

Ruhs 2011 Labor migration 2009 (p. 5) 46 high- and middle income countries (all OECD besides L and IS). (p. 48)

Aggregated oppeness index accessible (Table A3).

Thielemann 2003 Asylum 1985-1999 (p. 1) 20 OECD countries (US, CAN and AUS are not included) (p. 1)

Not publicly available

Timmer & Williams 1998; Timmer & Williamson 1996

Labor, control, some family reunif ication (historical analysis, probably there w ere no other f ields)

1860-1930 (p. 743) 6 countries: Argentina, Australia, Brazil, Canada, US, UK (p. 743)

Accessible: Appendix C (1996)

40

Table 3: Conceptualization

Name of Index Concept Definition Hierarchical Structure of Concept Redundancy ConflationCerna 2008 Yes, p.11. But definition of high-skilled immigration

restricted to Science and Technology sectors, omitting other HSI categories.

2 dimensions: Admission mechanisms (external) and w ork rights (internal), w ith three items each (see p.13)

No conceptual redundancy No conflation. Focus on policy output only.

Givens/Luedtke 2005

Yes, pp.3-6. 2 dimensions: Control/Integration dichotomy and restrictiveness of the law regarding immigrant rights (p.3-4).

No conceptual redundancy No conflation. Focus on policy output only.

Hatton 2004 Yes, pp.19-20. 4 dimensions: Access; procedure; outcomes; conditions, w ith 2 items for the f irst dimension and three each for the remaining (although a more clear statement w hich item goes into w hat dimension w ould have been desirable). (p. 24)

No conceptual redundancy No conflation. Focus on policy output only.

Klugman/Pereira 2009

No definition; only choice of sub-fields indicated w ithout justif ication (p.1-2).

Questions grouped into three different categories (admission; integration; enforcement) but no hierachical structure specified. (p. 4)

No redundancy. No conflation. Focus on policy output only.

Lowell 2005 Yes, p. 2. Not specif ied (only one- dimensional w ith 7 items, but used to classify programs for tw o dif ferent groups: temporary and permanent (pp.8-9)).

No conceptual redundancy No conflation. Focus on policy output only (although one could argue about 'enforcement mechanisms' being implementation see p.7)

Mayda 2005; Mayda & Patel 2004

No Not specif ied Unclear Unclear

Migrant Accessibility Index (part of the Global Migration Barometer 2008)

Yes, (but very brief) (p.2) Not specif ied. Some degree of conceptual redundancy (e.g. 'Ease of family reunif ication' is at a low er level of conceptual abstraction than 'government policy tow ards migration' and could have been subsumed under the latter).

Potential conflation by including "'de jure' and 'de facto' discrimination" in the same item. Also 'pow er of trade unions' is rather a context variable than part of the concept.

Ortega & Peri 2009

No Not specif ied but 3 dif ferent indices: Entry of non-asylum immigrants, Entry and/or stay of non-asylum immigrants, entry and/or stay of asylum seekers, w ith 3 categories each. How ever there is a residual category of law s that could not be classified. (p. 8)

Unclear No conflation. Focus on policy output only.

Oxford Analytica 2008

Yes, (p. 3-4; 7-9) Overall index w ith 2 dimensions: Administration and entry mechanisms, and Migrant Worker entitlement. Each further divided into tw o subdimensions (so called macro indicators) w ith 3-4 micro indicators, sometimes further split into sub-categories (pp. 17-18; 20).

Potential redundancy betw een "employment f lexibility" in the "w ork permit entitlement" macro-indicator and "employment rights" in the "employment and social rights" macro-indicator

No conflation. Focus on policy output only.

Pham & Van 2012

Yes (but very brief) (p.3.). Differentiate betw een 4 dif ferent 'tiers' of law s, conceptual hierachy of the four not further specif ied. (p. 7-8)

N.A. as only counting number of law s Conflation of output w ith implementation (e.g. "law enforcement")

Ruhs 2011 Yes (but very brief), (p.2) 2 separate indices: oppennes to labor immigration and migrant w orkers' legal rights. The latter has f ive dimensions: Civil/political, economic, social, residency and access to citizenship, and family reunif ication rigths. (p. 1)

Some degree of conceptual redundancy (e.g. R7 'equal pay', and R8 'equal employment conditions' might measure the same.)

No conflation. Focus on policy output only.

Thielemann 2003 Yes, (p.14-15). Distinguishes betw een three sets of instruments (access control, determination procedure, integration measures), w ith the first 2 having 1 item each, and the last 3 items. The relationship betw een the 3 instruments is not explained. (p. 15)

No redundancy. Conflation of output w ith outcome by including acceptance rates for subsidiary protection (p. 19-20).

Timmer & Williams 1998; Timmer & Williamson 1996

No explicit def inition, but they mention the outcome/output dif ference (1996: pxiii) also, they acknow ledge that there is a difference betw een restrictiveness and openness "policies often took the form of a large drop in or even disappearance of immigrant subsidies rather than outright exclusion of immigrants" (1998: p. 739)

Not specif ied. Some degree of conceptual redundancy (e.g. "subsidized or assisted passage" and "immigration treaties or contracts w ith shipping companies")

Conflation of output w ith implementation (e.g. -3 is assigned to countries w itch have "unenforced selectivity law s", w hile -5 ist assigned to countries w hich have "Closed doors, enforced" 1998: p. 741)

41

Table 4: Measurement

Name of Index Number of items Type of sources Measurement level Justification of measurement level

Coding rules Indication of sources

Coders Reliability tests Disaggregated data Availability of data

Cerna 2008 6 items (p. 26-27) Immigration law s collected via w ebsites, national authorities and international reports (p.12)

Ordinal items (0 to 3) (p.13-14)

No (available upon request (p.13))

Clear but partly based on subjective evaluations (p.26-27)

Yes (p.12) project researcher No Only tw o subindices provided (p.15-16)

Not publically available.

Givens/Luedtke 2005 Unclear Immigration law s (p.10) Ordinal, f irst dimension 3 steps (0, 1, 2) and second dimension 5 steps (1 to 5) (p.10)

No Not clear, as largely based on subjective evaluations (p.10)

Yes (p.11) Three research assistants (p.10)

Yes, but results not reported (p.10)

Not provided Not publically available.

Hatton 2004 11 items (p. 58-59) Immigration law s collected from international reports and secondary literature (p.59)

Nominal (0/1) (p.30-31, 58)

No Not clear, as largely based on subjective evaluations (p.30)

Yes (p.59) No information No Not provided Not publicly available

Klugman/Pereira 2009 84 questions, some are open comments questions (p. 22-27)

Immigration law s and experts' opinions, questionnaire sent to IOM country-level staff, w ebsites national authorities and legal consultancies, secondary literature and press articles (p.4-5)

Various nominal and ordinal items (p.22-27)

No Clear but partly based on subjective evaluations (p.22-27)

No Experts and project reserachers (p.4-5)

Yes, but results not reported (p.5)

Not provided Not publicly available

Lowell 2005 7 items (p. 7) No information Ordinal (1 to 4) (p.7-8) No Not clear, as largely based on subjective evaluations (p.7)

No No information No Not provided Not publicly available

Mayda 2005; Mayda & Patel 2004

No number of items specif ied/equal to the amount of law s/law changes. The indices are not build by several items but w ith a value (more/less restrictive) per change in the law /per new law (see w hole Mayda & Patel 2004)

Immigration law s collected via w ebsites national authorities, secondary literature and international reports (2004, entire document)

Ordinal (-1, 0, 1) (p. 24) No Clear and replicable (p.12, 24)

Yes (2004, last pages)

No information No Not provided Appendix on changes, but scored and aggregated data not available

Migrant Accessibility Index (part of the Global Migration Barometer 2008)

8 items (p. 23-26) Expert survey and ow n sources Ordinal (1 to 3/5) No Not clear, as largely based on subjective evaluations (Appendix 2)

Yes (p.3, appendix 2)

Project analysts No Not provided Not publicly available

Ortega & Peri 2009 No number of items specif ied/equal to the amount of law s/law changes. The indices are not build by several items but w ith a value (more/less restrictive) per change in the law /per new law (p.7)

Immigration law s (updated Mayda and Patel 2004) and Social reforms (Fondazione 2007) (p. 7-8)

Ordinal (-1, 0, 1) (p. 8) No Clear but largely based on subjective evaluation (p. 8)

Yes (p.7-8) Three research assistants and the tw o authors (p.8)

Yes, but results not reported (p.8)

Not provided Not publicly available

Oxford Analytica 2008 30 items (p. 20-32) Immigration law s collected via w ebsites, immigration authorities and legal consultancies, questionnaire sent to immigration authorities and research institutes (p.19)

Ordinal items (0 to 2/4/8) (p.21-32)

Only general rules indicated (p.16)

Clear but partly based on subjective evaluations (p.21-32)

Yes (p.46-100) No information No Disaggregated data provided (p.46-100)

Data available in appendix.

Pham & Van 2012 No number of items specif ied/equal to the amount of law s/law changes. They differentiate betw een negative and positive law s

Immigration law s collected by National Conference of State Legislatures and various advocacy organizations, new s databases (p.5-7)

Ordinal (-4 to +4) (p.7-8)

No Not clear, as largely based on subjective evaluations (p.7-8)

Yes (p.5-6) No information No Not provided Not publicly available

Ruhs 2011 35 items (p. 66-68) Immigration, labor and constitutional law s and secondary literature (p.16, 22)

Nominal (0/1) and ordinal (0 to 2/3) (p.16, appendix 2)

No Clear and replicable No Five roject researchers (p.22)

Yes, but results not reported (p.23)

Only one subindex provided (appendix 1)

Aggregated oppeness index accessible (Table A3).

Thielemann 2003 5 items (p. 18-21) Immigration law s collected via international reports (p.18)

Nominal (0/1) (p.19-21) No Clear and replicable (18-21) Yes (p.18) Author (p.18) No Not provided Not publicly available

Timmer & Williams 1998; Timmer & Williamson 1996

~20, but hard to count because of conflation, also in the legal sources many more items are mentioned (p. 763-769)

Immigration law s collected via secondary literature (1996: p.xvi-xxxi)

Ordinal (-5 to +5) (1998: p.740-741)

No Not clear, as largely based on subjective evaluations

Yes (1996: p.xvi-xxxi)

No information No Qualitative data provided (1996: p.xvi-xxxi)

Accessible: Appendix C (1996)

42

Name of Index Aggregation level Justification of aggregation level

Aggregation rule Justification of aggregation rule Test of aggregation rule /robustness check

Documentation of aggregation

Cerna 2008 Highest and second highest level of aggregation (p. 14).

No Additive, equal w eighting principle/unw eighted. Index going from 0-100 (going to most restrictive) and ranked (p. 13-14)

No justif ication. No Not provided.

Givens/Luedtke 2005 … … … … … …

Hatton 2004 Highest and second highest level of aggregation.

No Additive, unw eighted. Ranging from 0-11 (p. 31)

No justif ication. No Not provided.

Klugman/Pereira 2009 Highest level of aggregation (overall openness to international immigration score). Additional analyses of some of the individual items.

No No information available. No justif ication. No No information available.

Lowell 2005 Highest level of aggregation (one single score) for each of the tw o (temporary and permanent) indices.

No Additive, unw eighted and ranked from 0-100 (w ith the most restrictive country in the sample receiving a 100) (pp.8-9)

No justif ication. No Not provided.

Mayda 2005; Mayda & Patel 2004 … … … … … …

Migrant Accessibility Index (part of the Global Migration Barometer 2008)

Highest level of aggregation (one single score). No justif ication of aggregation level.

No Scores are converted to a 0-100 scores, and a w eighted average is taken (p. 6).

Justif ication of w eighting scheme: Based on several experts subjective evluation (p. 6).

No Information on the w eight of each item is availble (appendix 3) together w ith the total scores (p. 8), but information on the intermediate steps is not available.

Ortega & Peri 2009 … … … … … …

Oxford Analytica 2008 Highest and second highest level of aggregation.

No Additive, unw eighted at the second highest level. These scores are then normalized and a w eighted average is calculated at the highest level (p. 18). Conceptual structure is specif ied at 5 levels (see conceptualization table) but aggregation only explicit at highest level (overall score) and w hat ends up being the second highest (macro indicator) level, as the intermediate levels "fall out"

Justif ication of w eighting scheme: To give more prominence to programs w ith high inf low s of migrant w orkers (p. 18), based on the latest inf low data.

No Aggregation rule and aggregated data available (p. 18, 46-100). Weights not available.

Pham & Van 2012 … … … … … …

Ruhs 2011 Highest and second highest level of aggregation.

No Additive, unw eighted. Normalization of scores (0-1) before addition (21-22).

Justif ication of w eighting scheme: Transparency and simplicity, diff icult to justify w eights (p.21)

No Descriptive statistics (Table A2) and aggregated index (Table A3) available.

Thielemann 2003 Highest level of aggregation (one single score).

No Additive, unw eighted ranging from 0 to 5 (p.18, 21).

Justif ication of w eighting scheme: Same deterrence effect. Not theory based (p.21).

No No information available

Timmer & Williams 1998; Timmer & Williamson 1996

… … … … … …

43

Table 5: Aggregation and weighting