equilibria in an epistatic viability model under arbitrary strength of selection

J. Math. Biol. (1993) 31:397-410 dournalof

Mathematical 6101o9y

© Springer-Verlag 1993

Equilibria in an epistatic viability model under arbitrary strength of selection

Sergey Gavrilets N.I. Vavilov Institute of General Genetics, 3 Gubkin Street, Moscow B-333 117809, Russia

Received July 5, 1991; received in revised form January 23, 1992

Abstract. A class of viability models that generalize the standard additive model for the case of pairwise additive by additive epistatic interactions is considered. Conditions for existence and stability of steady states in the corresponding two-locus model are analyzed. Using regular perturbation techniques, the case when selection is weaker than recombination and the case when selection is stronger than recombination are investigated. The results derived are used to make conclusions on the dependence of population characteristics on the relation between the strength of selection and the recombination rate.

Key words: Epistasis - Stability of equilibria - Linkage disequilibrium

Introduction

The equations describing multilocus dynamics with account of selection and recombination are very complex and cannot be investigated without simplifying assumptions. To simplify the analysis one usually uses two approaches. The first approach includes the use of viability models that can be solved completely. However, the number of such models is small. So far, the multilocus dynamics is only clear for additive, multiplicative, and symmetric models (Ewens 1979). The second approach has been to make the assumption that the genetic system is in linkage equilibrium. This assumption, which results in the significant simplification of dynamic equations, is often applied for analyzing the problem of the maintenance of polygenic variability under stabilizing selection (e.g., Hastings and Horn 1990, Nagylaki 1989, Zhivotovsky and Gavrilets 1992). In general, linkage equilibrium holds approximately if selection is much weaker than recombination. Let there be two loci, r be the recombination rate, and s characterize the strength of selection. Then the errors in the predictions of steady gamete frequencies based on this assumption will have order of O(s/r) (Nagylaki 1976). This means that the errors are small if, roughly, s/r <~ 0.1. For unlinked loci (r = 0.5) it follows that the errors are small if s ~< 0.05. For linked loci the

Present address: Division of Environmental Studies, University of California, Davis, CA 95616, USA

398 s. Gavrilets

value of s has to be even smaller. It is quite probable that the selection in natural population is, in general, weak. At the same time, new experimental data give evidence that stronger selection is not just an exotic exception (Endler 1986). In this context it is important to analyze the behavior of multilocus systems when selection is not weak and, therefore, linkage disequilibrium cannot be neglected a priori. There are many interesting biological problems in this field. For example, let the assumption that selection is weak be used to prove that polygenic variability is maintained in a quantitative genetic model. Will this conclusion be valid if selection is not really weak? If yes, how do quantitative characteristics of the population (e.g., genotypic variance and mean heterozygosity) depend on the strength of selection and recombination rate? What are qualitative and quantitative consequences of an increase in the strength of selection?

This paper has two motivations. On the one hand, I continue the systematic study of the behavior of multilocus systems under selection described by a new class of viability models (Gavrilets, unpubl., Zhivotovsky and Gavrilets 1992). On the other hand, in the framework of this class I try to analyze the dependence of properties of quantitative genetic models on the relation between the strength of selection and the recombination rate. In this paper, I consider two-locus two-allele models. The structure of this paper is as follows. In the next section, a class of viability models that generalize the standard additive model for the case of pairwise additive by additive epistatic interactions is described. Then I present results on existence and stability of steady states that are characterized by linkage equilibrium. In the following section, the properties of steady states characterized by linkage disequilibrium are examined. Using regular perturbation techniques, I consider separately the case when selection is weaker than recombination and the case when selection is stronger than recombination. In the final section, I summarize main findings.

The model

Let there be n loci with two alleles each: A i and ai (i = 1 . . . . , n). Let the indicator variable li(l~) equal to 1 or 0 if the allele at i-th locus of paternal (maternal) gamete is A i or a~. Then the genotype of an individual is defined by the pair of vectors l = ( l , . . . , ln) and l ' = ( l ' , . . . , l~,). Denote by w++. the fitness (viability) of a genotype formed by gametes l and l ' and consider a class of viability models in the form of:

wtt.=1~+~[a~(l~+l~)+2b~lil~]+ c~/(l~+l;)(lj+J~), (1) i i ¢ j

where #, a¢, bi, and c~j = c/~ are constants (i ¢ j ) such that W~r/> 0 for all genotypes. This class of viability models was introduced in Zhivotovsky and Gavrilets (1992). Note that fitness w+~, remains unchanged as one exchanges vectors l and l' or indicator variables l~ and l~. We thus assume that the cis-trans effects (Turelli 1982) and the effects of sex are absent.

Model (1) includes many quantitative genetic models. In particular, fitness function (1) may arise as a result of "quadratic" stabilizing selection on a set of additive quantitative traits in the presence of overdominance, genotype-environ- ment interaction, pleiotropy or epistasis (Zhivotovsky and Gavrilets 1992). For illustrative purposes, consider a model of an additive trait z, assuming that the

Arbitrary strength of selection 399

contribution of allele Ai equals to ~/2 and that of allele ai equals to - ~ / 2 . In this notation:

n

z = ~ c~(l~ + l~ -- 1). (2) i

Let us describe stabilizing selection by a quadratic fitness function:

w(z) = 1 -- s(z - 0) 2, (3)

where 0 is an optimum phenotype, and s > 0 is the parameter measuring the intensity of selection.

Substituting (2) in (3) and rewriting the resulting equation in a form similar to (1), one can find the equations that express parameters/~, ae, b~ and c~j in terms of the parameters of the quantitative genetic model:

# = 1 - s(nc~ + 0) 2,

a i = s~2(2n -- 1 + 20/~), (4)

b i = - s o ~ 2, Cij = - - s a 2.

In this paper, I shall analyze the behavior of two-locus systems with account of recombination and viability selection described by (1) with a, = a2 = a, b~ = b2 = b, c12 = c2~ = c. The corresponding fitness matrix is presented in Table 1. Note that if a + b + 2c = 0, then this matrix is reduced to that of the so-called symmetric model (Table 2) investigated by Wright (1952) (see also Bodmer and Felsenstein 1967).

Consider a diploid randomly mating population with nonoverlapping gener- ations. Assume that the size of the population is sufficiently large so the effects of drift can be neglected. Let Xl, x2, x3, and x4 be the frequencies of gametes A I A 2 , Ala2 , a lA2, and ala2, and Pl, ql , P2, and q2 be the frequencies of alleles A~, al , A2, and a2, respectively. Denote by wjk the viability of a genotype formed by gametes j and k (j, k = 1 . . . . ,4). Then the changes in gamete frequencies are described by:

Axj = [(wj -- r~)xj + 6jrWl4D]/#, j = 1 , . . . , 4, (5)

Table 1. The fitness of genotypes

AIA1 A l a l a la l

A 2 A 2 m + 4a + 4b + 8c m + 3a + 2b + 4c m + 2a + 2b

A2a 2 m + 3a + 2b + 4c m + 2a + 2c m + a

a2a 2 m + 2a + 2b m + a m

Table 2. The fitnesses of genotypes in the case when a + b + 2c = 0. The values in the brackets correspond to the notation of Bodmer and Felsenstein (1967)

A~A~ A~a I a l a ~

A 2 A 2 # ( = I - - 6 ) # + a ( = l - 7 ) # - 4 c ( = l - ~ ) A2a 2 # + a ( = l - 7 ) # + 2 a + 2 c ( = I ) # + a ( = l - y ) a2a 2 # - - 4 c ( =1 -ct) # ÷ a ( =1 - y ) # ( =1 - 6 )

400 S. Gavrilets

where r is the recombination rate, wj = ~k WjkXk is the marginal fitness of gamete j, f f , = ~ w j x j is the mean fitness of the population, D =XlX4-X2X3, 61 = 6 4 = -- 1, 6 2 = 0 3 = 1. T h e quantity D is n a m e d the linkage disequilibrium coefficient and is used for characterizing the statistical association that arises between alleles at different loci. Together with D, one sometimes uses a normalized measure of linkage disequilibrium:

fl = DID . . . .

where Dmax=min(plq2, qlP2) when D > 0 , and Oma x =min(plp2, q~q2) when D < 0 (Lewontin 1964). The remaining part of this paper is devoted to the analysis of dynamic equations (5) and calculation of Pi, D, and/~ at stable steady states.

Steady states with D = 0

It follows from (5) that at a steady state with D = 0 either the frequency ofj- th gamete is zero, xj = 0, or the marginal fitness ofj- th gamete is equal to the mean

Table 3. Conditions for existence and stability of steady states with D = 0. Steady states are described by vectors of gamete frequencies. The conditions for existence and stability of states 3, 6 and 8 coincide with those of states 2, 5 and 7, respectively

Steady state Conditions Conditions for existence for existence and stability

Weak Strong selection selection

1 ( 1, 0, 0, 0) always

2 (0, 1, 0, 0) always

3 (0, 0, 1, 0) always

4 (0, 0, 0, 1) always

a+4c a÷4c ) a÷4c 5 2b , l + ~ f f - , 0 , 0 0 < - 2-~--

[" a+4c a+4c \ a+4c 6 ~ - - ~ , O, 1 "÷ -~f f - , O) O< Yg

( a ~bb) a 7 0 , - - 3 , 0 , 1 - 4 - 0 < - - 3 < 1

( a o 8 0 , 0 , - - 3 , 1 . 4 . 0 < - 3 < 1

a+2b+4c>O a+2b+4c>O a ÷ 2 b ÷ 3 c > O

a + 4 c < 0 a + 4 c < 0 a + 2 b > 0 a + 2 b > 0 2c>b 2c 0 a ÷ 2 b > 0 2c>b 2c 0 a ÷ 4 c > 0 a÷2b+4c<O a,4,2b.4.4c<O 2 c 0 a + 4 c > 0 a+2b+4c<O a ÷ 2 b ÷ 4 c < O 2 c 0 a > 0 a ÷ 2 b < 0 a ÷ 2 b < 0 2 c 0 a > 0 a + 2 b < 0 a + 2 b < 0 2 e < b < 0 e < b < 0

fitness of the population, wj = ~. The number of such steady states is eight (Table 3). Among them there are four states with a single gamete and four states with two gametes. The former four states exists always. The conditions for existence of the later four states are given in Table 3. The conditions for stability of a steady state can be found by linearizing system (5)-at the equilibrium point and calculating eigenvalues of the corresponding matrix. The exact conditions for stability of the steady states with D = 0 are presented in Appendix 1. Table 3 gives the conditions for existence and stability of these steady states when selection is "very weak" and is "very strong" relative to recombination. The meaning of the notions "very weak" and "very strong" in the present context is clarified in the next sections.

Steady states with D ~ 0

There are two types of steady states of (5) with D # 0: states with x 2 ~ x 3 and states with x2 = x3. The equilibrium values of gamete frequencies at the former states can be calculated in explicit form. However, these states do not exist when selection is very weak and are unstable at least when selection is very strong (see below). Therefore, these states are not considered here. As indicated by R. B/irger, a sufficient condition for the absence of stable steady states with x2 ~ x3 is c > b + r(# + 2a + 2c)/2. This follows from the consideration of (x2 + Ax2)/ (x3 + Ax3) - x2/x3. The equilibrium values of gamete frequencies at the symmetric steady states with x2 = x3 are defined by solutions of a polynomial of fourth order. In the next two sections, I present the results of the approximate study of these states.

Weak selection

Zero order analysis Let the selection be much weaker than recombination. Then the changes in allele frequencies can be approximated by the general relation (e.g., Wright 1935, Barton and TureUi 1987):

dpi Plqi~ ln - (6)

dt 2 OPi

Zhivotovsky and Gavrilets (1992) determined conditions for existence and stability of steady states of (6) in the case of fitness function (1) with arbitrary number of loci, n. If n = 2, there may be four steady states with single gametes, four steady states with two gametes, and a single steady state with four gametes. The conditions for existence and stability of the former eight states are presented in Table 3 in the column entitled "weak selection." The state with four gametes at which equilibrium allele frequencies are:

a

P~ =P2 = P * - 2(b + 2c) ' (7)

exists if

a 0 < - - < 1, (8a)

2(b + 2e)

402 S. Gavrilets

and is stable if

a + 2 b a>O, b < - a / 4 , b / 2 < c < - ~ - - - (8b)

In particular, if parameters of fitness function (1) correspond to the stabilizing selection (3) acting on an additive trait (2) (see (4)), then the number of polymorphic loci at a stable steady state can only be equal to one or zero.

First and second order perturbations In this section, using regular perturbation techniques I calculate characteristics of polymorphic steady states when selection is not very weak. Let # = #o + e#', a = ~a', b = ~b', c = ~c', where ~ is a small parameter and the coefficients #o, # ' , a', b', c" are 0(1) as e - >0. Assume that the probability of recombination, r, also is 0(1). Clearly the smaller the value of e is, the weaker will be the strength of selection relative to recombination. I shall look for steady gamete frequencies as power series:

xj = xi, o + exj,1 + ~2xj,2 +" ' ", (9)

where as the zero order terms I shall use the equilibrium gamete~frequencies from the preceding section, i.e., xl,o = (p.)2, X2,o = X3,o = p * ( 1 - p * ) , x4,0 = (1 __p,)2. Note that the corresponding perturbations for other characteristics of the population (such as allele frequencies, linkage disequilibrium, normalized linkage disequilibrium or the mean fitness) can be represented using perturbations for gamete frequencies (see Appendix 2, Eqs. (A1)-(A3)) . Zhivotovsky and Gavrilets (1992) presented the algebraic equations for perturbation terms up to the second order for an arbitrary number of loci. If n = 2, the perturbations for the polymorphic equilibrium are as follows. The first order perturbation for D is:

D,1 = 2c'p~q~/(#or). (10a)

Similar results have been obtained in a number of studies (Barton 1986, Hastings 1986, Turelli and Barton 1990).

The first order perturbation for allele frequency is:

2(c')poqo(Po - qo) P,1 = (b + 2C)#o r (10b)

The second order perturbation for linkage disequilibrium is:

D,2 = - ( 2 a ' + 2c') + (qo - P o ) 2 (11) r b ' + 2 c '

The eqs. (10-11) can be used to determine various characteristics of a polymorphic equilibrium. For example, the mean fitness of the population is:

= #o + g(#' + 2poa') + e22c'D,1 + O(e3). (12)

The residual term can be neglected if e 3 ~ 1. Therefore estimates of # based on zero, first and second order terms are valid for stronger selection (roughly, with e ~<0.45 ~ (0.1) 1/3) than those based only on zero order terms (roughly, with e < 0.1). Calculating perturbations of higher orders, one can find estimates of population characteristics that are valid for more and more strong selection. As a limit, one will obtain estimates that are valid for e ~< 1. However, the behavior of genetic systems for e > 1 remains unknown. This behavior is investigated in the next section.

Strong selection

Zero order analysis If selection is very strong relative to recombination, one can consider (5) assuming that r = 0 (Karlin and McGregor 1972). Then (5) can be rewritten as:

~ x j = (wj - ~ ) x j / ~ . (13) Without mutations four gametes can be considered as four alleles at a single locus. Methods for studying the dynamics of one-locus multi-allele populations are well known (e.g., Ewens 1979). In our case, the total number of possible steady states of (13) is fifteen. Among them there are four states with single gametes, six states with two gametes, four states with three gametes, and a state with four gametes. It follows from (13) that the marginal fitnesses of the gametes that are presented at a steady state are equal to ~ the mean fitness of the population, wj = ~. The consideration of the algebraic system W l - w2 = 0, W l - w 4 = 0, w 2 - w3 = 0 shows that this system does not have a solution. This means that the steady state with positive frequencies of all four gametes does not exist. The conditions for stability of other steady states can be found by linearizing system (13) at equilibrium point and calculating eigenvalues of corresponding matrix. The conditions for existence and stability of four states with a single gamete and four states with two gametes that are characterized by linkage equilibrium (i.e., with D - x l x4 - - X 2 X 3 = 0) are presented in Table 3 in the column entitled "strong selection." One also can show that two states with three gametes at which x2 = 0 or x3 = 0 can exist but cannot be stable. Note that at these states allele frequencies Pl (=~xl + x2) and P2(--- - )q q-X3) a r e not equal. This means that "unsymmetric" states are unstable. The conditions for existence

Table 4. Conditions for existence and stability of equilibria with D ¢ 0. Steady states are described by vectors of gamete frequencies. The conditions for existence and stability of state • correspond to the case of very weak selection, while those of states 1-4 correspond to the case of very strong selection

N Steady states Conditions Conditions for existence for existence and stability

a a+2b+4c a a * - O< - - < 1 a > O , b < - a / 4

2(b+2c) ' 2(b+2c) ' 2(b-4-2e)' 2(b+c) b < 2c

a + 2b + 4c~ 2 - ~ 2c) I ] a+ 2b +4c <O

a + c a + 2 b + 3 c ) a + c 1 -2(b+c~---~,0,0, ~ ( ~ .j 0< 2 ( b + c ~ < l c >O ,a+2b+3c <O

b + c < O , a + c > O

2 (0, 1/2, 1/2, 0) always a > 0, b < c < 0 a + b + c > O , a + b + 3 c < O

( ) a a a a + b + c 0 < 2 ( b + c ~ < l a > O , b < c < O 3 0, 2 (b+c ) ' 2 (b + c ) ' b + c a + b + c < 0

a + b + 3 c a+2b+4c a+2b+4c ) a+2c 4 b + c ' 2(b+c) ' ~ - + ~ ,0 0 < 2 ( b + c ) < l a > O , b < c < O

a +2b +4c <0 a + b ÷ 3 c > O

404 S. Gavrilets

and stability of four remaining states, among which there are two states with two gametes and two states with three gametes, are presented in Table 4.

First order perturbations . To calculate characteristics of stable steady states when the assumption about zero recombination is not valid I again use regular perturbation techniques. Let r = qr ' where i/ is a small parameter and let coefficient r ' and parameters #, a, b, c have order O(1) as ~ ~ 0. Clearly, the smaller the value of t/ is, the stronger will be the strength of selection relative to recombination. I shall look for steady gamete frequencies as power series in q in the form of (9). As the zero order terms I shall use the equilibrium gametes frequencies presented in Table 4. Substituting such power series in (5) and equating terms that correspond to the same power of t/, one can find perturbations of various orders (see Appendix 2 for an example).

Consider the stable steady states with D ~ 0 (see Table 4). At the first state, D O > 0 and flo = 1. First order terms for fl and ~ are:

b 1 = - r ' w 1 4 c , ff:1= - 4 r ' w 1 4 D o . (14a)

At the second state, P0 = 1/2, Do = - 1 / 4 , to = - 1 , and first order terms for fl and ~ are:

r:w14c fll -- ff~l = 2r 'w14Do. (14b)

4(a + b + 3c)(a + b + c) '

At the third state, Do = _p2 < 0, Po < 1/2 and rio = - 1. First order terms for fl and # are:

fll : r 'w14/c , 1# 1 : r ' w14Do(a -~- b + 3c)/c. (14c)

Finally, at the fourth state Do = - q o 2 < 0 , p o > 1/2, rio = - 1 , and

fll = r 'w14/e , ff:l = - - r ' W l 4 D o ( a --k b + c)/c. (14d)

The Eqs. (14) can be used to find characteristics of the population at a polymorphic equilibrium. The resulting estimates will have errors O012). Calcu- lating perturbations of higher orders, one can find estimates of population characteristics that are valid for more and more weak selection. As a limit, one will obtain estimates that are valid for q ~< 1. Combining these estimates with the estimates from the previous section, one can make conclusions about the behavior of the model over all possible relations between the strength of selection and the recombination rate.

Main findings

In this section, I summarize conclusions that can be drawn from formulae presented above.

1. Steady states with different numbers of polymorphic loci cannot be simultaneously stable

Consider, for example, the steady state with two polymorphic loci under weak selection which is stable if conditions (8) are satisfied. This state cannot be

simultaneously stable with state 1 in Table 3 because the latter is only stable when a + 2b + 4c > 0. This state cannot be simultaneously stable with states 2, 3, 5 -8 because these states can only be stable when c < b/2. And finally, this state cannot be simultaneously stable with state 4 because the former is stable when a < 0. B2) similar arguments it is easy to show that conditions for existence and stability of the states with zero and one polymorphic locus under weak selection cannot be simultaneously satisfied. Since the existence and stability of a steady state under weak selection are the necessary condition for existence and stability of a steady state under strong selection (see Table 3), these conclusions are also valid under strong selection. Note, however, that results of Hastings and Hom (1990) show that if the contributions of the loci are not equal ( a l ¢ a2, b~ ¢b2) then the states with different number of polymorphic loci can be simultaneously stable provided selection is weak.

2. There is at most one stable steady state with two polymorphic loci. This state is in linkage disequilibrium which is positive (negative) provided c > 0 (e < O)

Under weak selection there is a single steady state with two polymorphic loci, at p* = - a / ( 2 ( b + 2c)). Under strong selection there are four such states (see Table 4) but one can show that the conditions for existence and stability of these states are not compatible. In the model considered there are no stable steady states with two polymorphic loci and D = 0. If polymorphism is maintained at both loci then the steady state has a certain level of linkage disequilibrium. This disequilibrium is of first order under weak selection and can reach the maximum level (with fl = 1) under very strong selection. Under weak selection, the sign of D (see Eq. (10a)) coincides with the sign of parameter c that measures additive epistasis. Under strong selection the sign of linkage disequilibrium is positive at state 1 where c > 0 and negative at states 2, 3 and 4 where c < 0 (see Table 4). As indicated by R. Biirger, for the continuous time version of (5) the condition c < 0 (c > 0) implies that the region D ~< 0 (D ~> 0) is positively invariant and contains all co-limits, and that all stable steady states satisfy D ~< 0 (D ~> 0). To see this one can consider the time derivative of Z = x2x3/xl x4 as in the Appendix of Bfirger (1989).

3. Steady states with zero or one polymorphic locus can lose their stability as selection becomes stronger

This follows from the fact that the existence and stability of steady state under weak selection do not guarantee the existence and stability of a state under strong selection (see Table 3 and Appendix 1). One can imagine a situation when an increase in the strength of selection will result in an increase in genetic variability. This will take place if the population is near a monomorphic state that loses its stability as selection becomes stronger.

4. I f the polymorphism is mainta&ed at both loci under weak selection, then both loci remain polymorphic as selection becomes stronger

This conclusion follows from the condition for existence and stability of the steady state (7) given in (8) and those presented in Table 4. Let the steady state

406 s. Gavrilets

with two polymorphic loci be stable under weak selection and let selection become stronger. Then one of the four states described in Table 4 is stable under strong selection. If c > 0 this state will be state 1. Now let c < 0 and let

= p * - 1 / 2 be the deviation of equilibrium allele frequency at weak selection from 0.5. Using conditions for existence and stability presented in Table 3, one can show that if V < - ( 2 ( 2 + b/c)) -~ then, as the selection becomes stronger, the stable state will be state 3. If 7 > (2(2 + b/e)) -1, then this state will be state 4. Finally, if allele frequencies p* under weak selection are sufficiently close to one half, ~ < (2(2 + b/c))- ~, then the stable state under strong selection will be state 2. Note that since b < c < 0, if 7 < - 1/6 or 7 > 1/6 (or equivalently, p* < 1/3 or p* >2/3) the conditions for existence and stability of states 3 or 4 are satisfied.

5. As the selection becomes stronger, the equilibrium allele frqueney shifts to (from) 0.5 provided the sign of D is negative (positive). The linkage disequilibrium can reach the maximum possible level

Under very weak selection, the equilibrium allele frequency is given by (7). From (10b) it follows that if d > 0, then the sign o f p coincides with that of (q* - p * ) , and equilibrium allele frequency p ( --p* + ep + O(e)) shifts from 1/2, to a first order. If D < 0, p shifts to 1/2. As selection becomes very strong, the equilibrium allele frequency shifts from p* to one of the four values presented in Table 3. Again one can show that p becomes closer (farther) to (from) 1/2 provided D < 0 (D > 0). The maximum possible shift o f p is just the difference between p* and the appropriate value P0. The disequilibrium is small under weak selection and can reach the maximum level (with/~ = 1) under very strong selection. Note that in the two-locus mutation-selection model analyzed by Hastings (1988) the disequilibrium can reach only half its maximum value.

6. The mean fitness of the equilibrium population increases as the recombination rate decreases

Under weak selection the mean fitness of the population is independent of recombination and linkage disequilibrium, to a first order. But to a second order, since c'D > 0 and D ~ r-~, the mean fitness increases as the recombination rate and absolute value of D decrease. For strong selection, as can be seen from (14) with account of the conditions for existence and stability presented in Table 3, ~ < 0 and #1 ~ r at all steady states. In other words, the mean fitness increases as the rate of recombination decreases, to a first order.

7. Even moderate selection is sufficient to generate linkage disequilibrium causing significant phenotypic effects

All the previous conclusions are qualitative ones. Now let us discuss effects of linkage disequilibrium from the quantitative point of view. Consider the model of an additive quantitative trait (2) under stabilizing selection (3) with n = 2. In this case b = - s ~ 2, c - - - s ~ 2, and if 0 = 0 , then a = 3 s a 2, so that a + b + 2c = 0. As was already stated, if a + b + 2c = 0, then the fitness matrix in Table 1 is reduced to that of the symmetric model (Table 2). In this symmetric

model there is a polymorphic steady state with allele frequencies equal to 1/2 and with linkage disequilibrium D = r i m - (1/4)x/1 + 16rZ/rn 2, where m = 6 - ~ in the notation of Bodmer and Felsenstein that is equal to m = -4c/(12 + 2a + 2c) in our notation. Assume further that ~2= 1. Then the exact expression for the normalized linkage disequilibrium is

B = ( r / s ) - + ( r l s ) 2 (15) (e.g., Bodmer and Felsenstein 1967). This steady state is unstable for r > 0, but serves as well as stable equilibrium to demonstrate the amount of linkage disequilibrium generated by selection (see also Bulmer 1974). In (15), fl also can be interpreted as minus the ratio, fl = - C L IV g, of two components of the genotypic variance, G = Vg + CL in the notation of Bulmer (1974). One of them is the genic variance Vg = ~2~p iq i and the other is the contribution of linkage disequilibrium Cr = ~2~ic~jDij. Note also that 1 + fl is the ratio, 1 + fl = G/Vg, of the genotypic variance actually observed in the population, G, to that existing in the population under very weak selection (when D and CL are equal to zero). The dependence of/~ on r/s is presented in Fig. I. To illustrate the degree of adequacy of solutions that can be found by regular perturbation techniques, this figure also shows two families of curves approximating (15) when selection is weaker (r/s > 1) and stronger (r/s < 1) than recombination. These curves were

-0.5

÷ .F

g ÷ ÷

g g'l'÷ . n ~

÷ / t/p~ "[' /,.d([ .~ I [ [ [~

- O~ (, I(%%(%

%

#

. • . . , "

Fig. 1. The numerical illustration of the dependence of normalized linkage disequilibrium, fl, on the ratio r/s. Exact solution (15) is shown by the solid line. Diamonds and rectangles describe the linear, fl = - (1 /2 ) s / r , and cubic, fl = - (1 /2 )s / r ( I /8 ) ( s / r ) 3, approximations for weak selection• Pluses and exes stand for the linear, fl = 1 - r / s , and cubic, fl = 1 - r / s - (1~2)(r / s ) 2, approximations for strong selection• Dots represent "the upper boundary" for/~ calculated from Hastings's estimates

% • %

• % Z vl 2

_1 I , I I I 0 1 3 4 5

r / s

408 s. Gavrilets

found by expanding (15) in power series as e = s/r <~ 1 and as rl = r/s ~ 1. First order estimates can also be calculated from Eq. (10a) for weak selection and from Eq. (14) for strong selection. One can see that linkage disequilibrium results in significant phenotypic effects even when the selection is not extremely strong. For example, fl = - 0 . 4 if r/s = 1. Figure 1 also shows "the upper boundary" for fl calculated from Hastings's (1986b) estimate D < Aw/(lOr), where A w is the maximum absolute value of the deviation of any fitness from that of the double heterozygote. In our case A w = 4s. Figure 1 shows that significant effects of linkage disequilibrium are quite compatible with Hastings's estimate which is "conservative" with non-weak selection.

Summarizing, increase in the strength of selection causes the following qualitative effects. Steady states with zero or one polymorphic loci can lose their stability what can cause an increase in the genetic variability. The steady state with two polymorphic loci keeps the stability. The allele frequencies shift to or from 0.5. This shift results in the increase or decrease of the genic variance or the mean heterozygosity. Linkage disequilibrium is present at any state with two polymorphic loci and can reach a maximum level as selection becomes stronger. Significant changes in steady population characteristics can be observed under reasonably strong selection.

Appendix

1. Exact conditions for stability of steady states with D = 0. These conditions were found by linearization of Eqs. (5) at steady states and

by calculation of corresponding eigenvalues using a computer algebra program. Below steady states are described by vectors of gamete frequencies.

(i) State (1, 0, 0, 0) is stable if:

- ( a + 2b + 4c) < 0, - 2 ( a + 2b + 3c) - rwl4 < O,

where w14 = m + 2a + 2c.

(ii) States (0, 1, 0, 0) and (0, 0, 1, 0) are stable if:

a + 4c < 0, - (a + 2b) < 0, 2(c - - b ) - - r W l 4 < 0.

(iii) State (0, 0, 0, 1) is stable if:

a < 0 , 2 ( a + c ) - r w l 4 < O .

(iv) States ( - ( a + 4c)/(2b), (a + 2b + 4c)/(2b), 0, 0) and ( - ( a + 4c)/(2b), 0, (a + 2b + 4c)/(2b), 0) are stable if:

- ( a + 4c)(a + 2b + 4c)/(2b) < O,

2a(b - c) - 4b 2 - b(2c + r) + 8c 2 +_ v / (c + r/2) 2 + cr(a + 4c)/b < O.

2b

(v) States (0, - a / 2 b , O, (a + 2b)/(2b)) and (0, - a / 2 b , (a x 2b)/(2b), 0) are stable if:

- a ( a + 2b)/(2b) < 0, a(1 - c/b) + c - r/2 + x/(2c - r) z - 4acr/b < O.

2. Calculation of the first order perturbations for strong selection. At all stable steady states with D ~ 0, allele frequencies at both loci are

equal, say to p. This means that the allele frequency p and linkage disequilibrium D can be represented as p = l / 2 - ( x 1 - x 4 ) / 2 and D = X l - p 2 - = x 4 - q 2 = p q - x2, and that Dmax is equal to p2 provided D < 0 and p < 1/2, to q2 provided D < 0 and p > 1/2, and to p q provided D > 0, respectively. Here Xl, x2 and x4 are frequencies of gametes A1A2, A l a z and ala2, q = 1 - p . From the formulae for D and p it follows that first order perturbations can be written as:

Pl = (xl,1 - - x 4 , 1 ) / 2 , D 1 : Xl, 1 - - 2pop1 = x4,1 - - 2q0ql. (A1)

Let D < 0 and p < 1/2. Then /~ = - 1 4- x l / p . Substituting xt and p by corresponding power series (9) and expanding the resulting equation into Taylor series, one can find perturbations/~i for normalized linkage disequilibrium/~:

flo = - 1 + Xl,o/Po, fll 2 l,o/Po, (A2) = x l , i / p o _ Z p l x 3 . . .

I f D < 0 and p > 1/2, then/~ = - 1 + xn /q 2, and the corresponding perturbations are:

flo = - - l "-k X 4 , 0 / q o , f l l : X 2 , 1 / q ~ - - 2 q l X 4 , 0 / q 3 , . . . ( a3 )

I f D > 0, then fl = 1 - x2 / (pq) , and

flo = 1 - Xz,o/(Poqo), 1~1 = - Xz,1/(Poqo) + 2plxz,0(q0 - Po)/(Poqo).

These formulae show that to calculate the first order perturbations for p, D and /~ one needs to know only some of the first order perturbations for gamete frequencies x i.

The equations for first order terms xj, have the form:

(Wj, 1 - - ]~I)Xj,0 "q- (Wj, 0 - - l~o)Xj , 1 "~ 6jr" w14D o = 0, (A4)

where wj,0 and wj,1 are zero and first order terms for the marginal fitnesses of gametes, #o and #1 are the corresponding terms for the mean fitness of the population. It follows from the results for one-locus multi-allele models that either Xj, o = 0 or Wzo = v~ 0. Hence only one of the two first terms is present in Eq. (A4). Consider, for example, state 3 in Table 4 at which xl,0 = 0. Calculating zero order term for wl, first order term for w4, and zero and first order terms for ~, one can show that:

Wl,0 - - W0 : 2c, w4,1 - wl = a(xl , l - x4,1) - 2cx1,1 .

Substituting these formulae into Eqs. (A4) with j = I and j = 4, one gets the equations for xl,~ and x~,~ -Xl,4:

2CXl,1 -[- r'w14Do = 0, (a(xl,1 - - X4,1) - - 2CX1,1)X4,0 q- r ' w 1 4 D o = O.

Applying formulae for Pl and/71 presented above, one can derive Eqs. (14c) of the main text. Equations (14) for first order perturbations at other steady states can be derived using similar approach.

Acknowledgements . I thank W. Ewens, T. Nagylaki, V. P. Passekov, R. Biirger, and M. Turelli for helpful comments on earlier drafts and suggestions.

References

Barton , N. H.: The effect o f l inkage and dens i ty -dependent regu la t ion on gene flow. Hered i ty 57,

4 1 5 - 4 2 6 (1986)

410 S. Gavrilets

Barton, N., Turelli, M.: Adaptive landscapes, genetic distance and the evolution of quantitative characters. Genet. Res. 49, 157-173 (1987)

Bodmer, W. F., Felsenstein, J.: Linkage and selection: theoretical analysis of the deterministic two locus random mating model. Genetics 57, 237-265 (1967)

B/irger, R.: Linkage and the maintenance of heritable variation by mutation-selection balance. Genetics 121, 175-184 (1989)

Bulmer, M. G.: Linkage disequilibrium and genetic variability, Genet. Res. 23, 281-289 (1974) Endler, J. A.: Natural selection in the wild. Princeton, NJ: Princeton University Press 1986 Ewens, W.: Mathematical Population Genetics. Berlin Heidelberg New York: Springer 1979 Hastings, A.: Multilocus population genetics with weak epistasis. II. Equilibrium properties of

multilocus selection: what is the unit of selection? Genetics 112, 157-171 (1986) Hastings, A.: Disequilibrium in two-locus mutation selection balance models. Genetics 118, 543-547

(1988) Hastings, A., Hom, C. L.: Multiple equilibrium and maintenance of additive genetic variance in a

model of pleiotropy. Evolution 44, 1153-1163 (1990) Karlin, S., McGregor, J.: Application of method of small parameters to multi-niche population

genetic model. Theor. Popul. Biol. 3, 186-209 (1972) Lewontin, R. C.: The interaction of selection and linkage. I. General considerations; heterotic

models. Genetics 49, 49-67 (1964) Nagylaki, T.: The evolution of one- and two-locus systems. Genetics 83, 583-600 (1976) Nagylaki, T.: The maintenance of genetic variability in two-locus models of stabilizing selection.

Genetics 122, 235-248 (1989) Turelli, M.: Cis-trans effects induced by linkage disequilibrium. Genetics 102, 807-815 (1982) Turelli, M., Barton, N. H.: Dynamics of polygenic characters under selection. Theor. Popul. Biol. 38,

1-57 (1990) Wright, S.: Evolution in populations in approximate equilibrium. J. Genet. 30, 257-266 (1935) Wright, S.: The genetics of quantitative variability. In: Mather, K. (ed.) Quantitative Inheritance, pp.

5-41. London: Her Majesty's Stationary Office 1952 Zhivotovsky, L. A., Gavrilets, S.: Quantitative variability and multilocus polymorphism under

epistatic selection. Theor. Popul. Biol. 42, 254-283 (1992)

equilibria in an epistatic viability model under arbitrary strength of selection

Documents