\documentstyle{report} \textwidth 3in %was 2.7in \textheight 10in \topmargin -35pt \renewcommand{\baselinestretch}{.83} %\documentstyle{report} %\textwidth 237pt %%\textwidth 244pt was too wide %\topmargin -85pt %\textheight 750pt %\renewcommand{\baselinestretch}{.86} %\begin{document} \begin{document} \sloppy %Coalescent time in the presence of inbreeding \noindent{\bf Introduction.} The genetic structure of a population depends on the ancestry of the genes in the population, in particular the extent to which they have common ancestry. The entire ancestral pedigree of all the genes in a population is called the coalescent. Sometimes the term coalescent is used to refer to the time since a common ancestor of the entire population, or the time since a common ancestor of the two genes in an individual. The coalescent is well defined for all populations, for all of the above definitions. Kingman (1982) presented the first analysis of the coalescent. He determined the distribution of the number of ancestral genes of the present population as a function of time for a random mating population with a Poisson progeny distribution. The distribution of the number of progeny is not critical to his analysis, but random mating is. Subsequent studies of the coalescent have incorporated selection (Kaplan, Darden, and Hudson 1988) and migration (Takahata 1988, Slatkin 1991), but random mating has remained an essential assumption of all analyses. Several methods have been used to model nonrandom mating, some based on phenotype (assortative mating), and others based on ancestry (inbreeding). Inbreeding has been modelled with finite pedigrees (Thompson 1986), regular systems of inbreeding (Wright 1921), and inbreeding coefficients (Cockerham 1970). Since finite pedigrees generally do not extend back to a common ancestor, regular systems of inbreeding and inbreeding coefficients are employed here. The time since a common ancestor of the entire population is conceptually essentially the same as the time until fixation of a gene, and in the case of a random mating population with Poisson progeny distribution it is the same. This allows us to use the diffusion approximation (Maruyama \& Kimura 1971) to approximate the time since a common ancestor for the entire population when inbreeding (e.g., selfing) is superposed on a random mating background. In particular, in the absence of selection the time until fixation of a mutant gene (conditioned on fixation) is given by $$ \int_{\frac{1}{2N}}^{1} \frac{2\xi (1-\xi)}{V_{\delta \xi}} d \xi + \frac{1 - \frac{1}{2N}}{\frac{1}{2N}} \int_{0}^{\frac{1}{2N}} \frac{2 \xi^{2}}{V_{\delta \xi}} d \xi $$ (Crow \& Kimura 1970, p.430), where $V_{\delta p}$ is the sampling variance of the change in allele frequencies. Other methods are employed to study the coalescent under regular systems of inbreeding. {\bf N.B.:} Autozygosity is used below in the sense of identity by descent. The Poisson progeny distribution is actually a binomial progeny distribution since the population size is constant; the difference between the Poisson and binomial progeny distributions will not significantly affect the results. \newpage \noindent{\bf Poisson progeny distribution.} For random mating with a Poisson progeny distribution $V_{\delta p} = p(1- p)/(2N)$, which produces the well known result that the expected time until fixation of a mutant gene is $4N$ generations, the same as the coalescent time found by Kingman. If a fraction $s$ of the population selfs while the rest of the population randomly mates (allowing the possibility of selfing in the random mating fraction), $V_{\delta p} = (p(1-p)/2N)(1+ (\frac{s}{2}/(1 - \frac{s}{2})))$. Hence the time until fixation is $4N/(1+(\frac{s}{2}/(1 - \frac{s}{2})))$. In particular, if the population is entirely selfing ($s=1$), the time until fixation is the $2N$, the same as for a population of half the size (heuristically, autozygosity is soon obtained, after which both genes in an individual are sampled as a unit). The quantity $ (\frac{s}{2}/(1 - \frac{s}{2})))$ is equal to the asymptotic probability of autozygosity in an infinite population which selfs $s$ of the time. Denoting with $f$ the probability of autozygosity due to the mating structure (as opposed to finite population size), it can be shown that if inbreeding is superposed on a random mating background with Poisson progeny distribution, the time until fixation is $4N/(1 + f)$. \\ \noindent {\bf Two progeny per individual.} When the population is constrained to two progeny per individual (one in the case of selfing), the sampling variance of the change in allele frequencies is $V_{\delta p} = p(1-p)/(4N)$ for a random mating population. This provides that the expected time until fixation is $8N$ generations, twice the value for a population with a Poisson progeny distribution. If selfing with frequency $s$ is superposed on the random mating population (which allows selfing), $V_{\delta p}$ becomes $(p(1-p)/(4N))(1 - (\frac{s}{2}/(1-\frac{s}{2})))$. This provides that the time until fixation is $8N/(1 - (\frac{s}{2}/(1-\frac{s}{2})))$. In the case of obligate selfing ($s = 1$), the time until fixation is infinite, which is obvious since a gene cannot spread beyond a single lineage. Nonintersection of lineages also assures that the time since a common ancestor is infinite. For values of $s$ less than one, the time until fixation increases with more selfing, which is opposite to the effect found in the case of Poisson progeny distribution. Analogous to the case of Poisson progeny distribution, $(\frac{s}{2})/(1- \frac{s}{2})$ is the asymptotic probability of autozygosity. It can be shown that the expected time until fixation is $8N/(1-f)$ for any mating structure superposed on random mating with two progeny per individual. \newpage \noindent {\bf Regular Systems of Inbreeding} The regular systems of inbreeding considered here have two progeny per individual, and half-sib and circular pair mating have asymptotic probability of autozygosity equal to one in an infinite population (the model of maximum avoidance of inbreeding is not defined for infinite populations). However, they are not superposed on a background of random mating, so the previous analysis is not valid. Half-sib, circular pair, and maximum avoidance of inbreeding (Kimura \& Crow 1963) all have the expected time since a common ancestor of the two genes in an individual equals $4N-2$, the same value as for a random mating population constrained to two progeny per individual. (Another measure of inbreeding is the amount of genetic identity within (homozygosity) and between individuals at equilibrium with mutation. These values depend on the mutation rate. Campbell (1993) found numerically that genetic identity was similar for random mating and avoidance of inbreeding, but there is substantially more homozygosity and less identity between individuals under half-sib and circular pair mating. Hence the time since a common ancestor of two genes in an individual of $4N-2$ generations does not reflect that the populations have the same genetic structure.) The difference equation analyses of the above regular systems of inbreeding characterize each pair of genes by their distance in the pedigree (i.e., the time since a common ancestor of the individuals which contain the genes). Under avoidance of inbreeding, the time since a common ancestor of two genes in an individual ($4N-2$) is the largest time for any pair of genes. For half-sib mating and circular pair mating, $4N-2$ is essentially the shortest time among all pairs of genes (pairs of genes in half-sibs and sibs have a slightly shorter expected time since a common ancestor). Of course, with random mating the expected time since a common ancestor is the same for all pairs of genes. Although the coalescent time for the entire population has not been calculated, this suggests that it should be greater for half-sib and circular pair mating than for random mating than for maximum avoidance of inbreeding. \newpage \noindent {\bf Discussion.} The preceding preliminary results provide initial insight into the effect of inbreeding on the coalescent. It is demonstrated that inbreeding can either increase or decrease the coalescent time depending on the progeny distribution. (This is related to the fact that selfing decreases both heterozygosity and the number of segregating alleles with a Poisson progeny distribution, but decreases heterozygosity and increases the number of segregating alleles when there are two progeny per individual.) The study of regular systems of inbreeding shows that the time since a common ancestor of two genes in an individual does not indicate what the coalescent time for the entire population is (although it does provide a lower bound). But the above results raise more questions than they provide answers. The diffusion approximation raises the question: exactly what is the difference between the time until fixation of a mutant gene and the time since a common ancestor of the entire population; are they equivalent? Since the inbreeding occurs in the diffusion equations in a factor which is independent of allele frequencies, is the structure of the coalescent unchanged, and only the time scale upon which it is measured altered? Since inbreeding reduces the coalescent time for the Poisson progeny distribution but increases the coalescent time with two progeny per individual, is there a progeny distribution for which the coalescent is unaffected by inbreeding; what is that progeny distribution? For regular systems of inbreeding, the basic question of calculating the coalescent time remains. Then the structure of the coalescent must be investigated for the above question of whether only the time scale and not the structure of the coalescent is affected by inbreeding. Also, regular versus random sib mating, first cousin mating, etc. should be juxtaposed. \vspace{10pt}\\ \begin{small} Literature cited:\\ Campbell, R. B. 1993. {\it Th. Pop. Biol. 43\/}:129--140.\\ Cockerham, C. C. 1970. pp.104-127 in K. Kojima, ed. {\it Math. Tpcs. in Popul. Genet.} \\ Crow, J. F. \& M. Kimura. 1970. {\it Intro. Pop. Gen. Th.}\\ Kaplan, N. L., T. Darden, \& R. R. Hudson. 1988. {\it Genetics 120\/}:819-829.\\ Kimura M. \& J. F. Crow. 1963. {\it Genet. Res. 4\/}:399--415.\\ Kingman, J. F. C. 1982. {\it J. Appl. Prob. 19A\/}:27-43. \\ Maruyama, T. \& M. Kimura. 1971. {\it Jap. J. Genet. 46\/}:407-410.\\ Slatkin, M. 1991. {\it Genet. Res. 58\/}:167-175.\\ Takahata, N. 1988. {\it Genet. Res. 52\/}:1213-222.\\ Thompson, E. A. 1986. {\it Pedigree Anal. in Hum. Genet.}.\\ Wright, S. 1921. {\it Genetics 6\/}:111-178.\\ \end{small} \end{document}