text: user8725: I am reading Section 5.2, Kinship and Inbreeding Coefficients, of Kenneth Lange, Mathematical and Statistical Methods for Genetic Analysis. There the kinship coefficient $\Phi_{i,j}$ is defined for two relatives $i$ and $j$ as the probability that a gene selected randomly from $i$ and a gene selected randomly from the same autosomal locus of $j$ are identical by descent.
  I would think $\Phi_{i,j}$ should depend on the particular population frequency of the gene or allele at that locus. However, the book does not seem to indicate that or at least does not stress that at all. Is there a dependence or not? The same question applies to the inbreeding coefficient.
  More importantly, I doubt this definition is mathematically rigorous.
  
  Here is an example to make what puzzles me clearer. It must be that the probability I am computing is not what the kinship coefficient. But this is the definition means to me. Some expert please elucidate the correct meaning of the definition.
  Consider the simplest pedigree tree of one family of two parents with brothers $A$ and $B$. Allele $a$ is detected at a particular locus. Let $b$ be the other allele of the same gene. We ask for the probability of the brother $B$ having allele $a$ conditioned on $A$ having allele $a$.
  In the following, to clutter symbols, we abuse the symbols by use $a$ to denote the population frequency or the unconditional probability of finding allele $a$ at the locus for the whole population as $a$. The same goes for allele $b$. $a,b\in[0,1]$ and $a+b=1$. We solve this problem by listing as follows all possible parental gene configurations, the configuration probability (considering the symmetry of paternal and maternal loci), as well as the probability finding allele $a$ for either $A$ or $B$ at the locus.
  \begin{align}
  \text{parent config}~~~~ & \text{Pr(config)} & \text{Pr($a$|config)} \\
  aa|aa~~~~~~~~~  &  ~~a^4   & 1 \\
  aa|ab~~~~~~~~~  &  4a^3b   & 1 \\
  aa|bb~~~~~~~~~  &  2a^2b^2 & 1 \\
  ab|ab~~~~~~~~~  &  4a^2b^2 & \frac34 \\
  ab|bb~~~~~~~~~  &  4ab^3   & \frac12 \\
  bb|bb~~~~~~~~~  &  ~~~~b^4 & 0 \\
  \end{align}
  The joint conditional probability of both $A$ and $B$ have allele $a$ at the locus is Pr($a$|config)$^2$. So the required conditional probability is
  $$P(r):=\text{Pr}(B\text{ has } a|A\text{ has }a)=\frac{\sum_\text{config} \text{Pr(config)Pr(a|config)}^2}{\sum_\text{config} \text{Pr(config)Pr(a|config)}} = \frac{r^3+4r^2+2r+4r(\frac34)^2+\frac4{2^2}}{r^3+4r^2+2r+4r\frac34+\frac42}, \quad r:=\frac ab\in [0,\infty).$$
  $P(r)$ depends on $r$ which depends on the population frequency of the alleles as I have claimed.
  Consider $[r^3+4r^2,2r,3r, 2]$ as a positive entry vector. $P$ is a convex combination of $v:=\big[1,1,1,\frac34,\frac12\big]$. So $P$ is between the minimum and maximum of the entries of $v$, and
  $$\frac12\le P\le 1.$$
  We see the extrema are achieved at
  $$P(r)=\begin{cases}
  \frac12, \quad r=0 \\[0.7ex]
  1,\quad r\to\infty
  \end{cases}$$
  
  user8725: The definition as set by Kenneth Lange's book is indeed quite vague and thus unrigorous. It is a probability not conditioned on observing any particular allele at a locus thus not on the population frequency of that allele, but conditioned on the following where we focus on one particular locus. It is rigorously defined as follows
  A graph of lineage contains nodes, which we call persons, of a particular locus. Each node, or person, contains exactly two genes. We draw one and only one directed edge (parental relationship) from a gene of a parent person to the gene of a child person if the latter is inherited from the former.  This is a directed acyclic graph (DAG). Each gene can be connected by no more than $1$ edge directed to (as opposed to away from) it. The two genes in one person cannot be connected by edges. Two persons are said to be connected if there is an edge pointing from a gene of one node to a gene of the other node. Two persons cannot be connected by more than one edge. For a pair of connected persons and their directed edge between them, the edge is equally probable to connect to one gene as the other gene of a person, i.e. the probability is $\frac12$. A pair of genes (persons) are said to be connected if there exists an undirected (simple) path from one gene (person) to the other. Two thus connected genes are called identical by descent.
  In this setting, the DAG with respect to the persons is given and deterministic whilst any directed edge in the DAG assigned to two persons are random with independent probability $\frac12$ with respect to the genes within each person.
  For a pair of persons on a lineage DAG that is connected, the probability of one gene equally probably chosen amongst the two genes of one person being connected, or identical by descent, to one gene equally probably chosen amongst the two genes of the other person is the coefficient of kinship. It can be proven that the probability is $\displaystyle\sum_{p\in P} \frac1{2^{n(p)}}$ where $P$ is the set of all distinct (simple) paths connecting one person (not gene) to to the other and $n(p)$ is the number of persons on person connecting path $p$.
  As an additional assumption, the genes that are connected have to be the same and thus assume the same allele.
  
  With regard to the conditional probability example in the question, the coefficient of kinship is not adequate to be used to compute the desired probability. Part of the reason is the question inquires about the existence of an allele rather than picking one gene with 1/2 of the probability. We need some more fundamental probability to compute that. We will demonstrate that as follows.
  
  We will compute the probability of the two siblings having both allele $a$ as follows.
  
  For a pair of full siblings, they can have exactly $0, 1, 2$ identical by descent genes. Let the respective probabilities be $r_0, r_1, r_2$. Notice we are not picking genes of a person at random but talking about the existence of particular types of genes in a person. For two full siblings, some thoughts lead to $r_0=\frac14, r_1=\frac12, r_2=\frac14$.
  By symmetry and without the loss of generality, we fix the connecting genes in the two siblings of the respective edges. We only need to consider the connections of the edges relative to the genes of the parents.
  (1) Conditioned on there is no identical by descent genes, the sibling genes can independently assume any allele. The probability of having the configuration of $ab|ab$ (not considering gene ordering) is $2^2a^2b^2$ considering the distinct ordering of the alleles, that of $aa|ab$ is $\binom41 a^3b$, and that of $aa|aa$ is $a^4$. The total probability is thus
  $$Pr_0:=\frac14(4a^2b^2+4a^3b+a^4).$$
  (2) When there is exactly $1$ pair of identical by descent genes, the two identical by descent genes can only assume the same allele, while the other genes can assume alleles independently. When the identical by descent genes assume allele $a$ the probability of the two siblings both having allele $a$ is $a$. When the two identical by descent genes assume allele $b$ the desired probability is then $ba^2$. The total probability is then
  $$Pr_1:=\frac12(a+ba^2).$$
  (3) When there are exactly $2$ pairs of identical by descent genes, each pair can only assume the same allele but independently of each other. The only way that the two siblings not to both have allele $a$ is for the two identical by descent pairs to assume allele $b$. The total desired probability for the two siblings to both have allele $a$ is then
  $$Pr_2:=\frac14(1-b^2)$$
  
  Now we compute the probability of any one of the siblings having allele $a$. Each gene can of course assume alleles independently. So the probability is simply
  $$Pr_3:=1-b^2.$$
  
  Therefor the desired probability of both siblings having allele $a$ given that one of them has allele $a$ is
  $$Pr(a):=\frac{Pr_0+Pr_1+Pr_2}{Pr_3}=\frac{\frac14a(a^3-6a^2+5a+4)}{a(2-a)}$$
  since $b=1-a$. Some tedious algebra which can be carried out by, say, Mathematica shows this is identical to the expression in the question in terms of $r=\frac{a}{1-a}$.
  $$Pr'(a)=\frac12\Big(2-\frac1{(2-a)^2}-a\Big).$$
  It is easy to see $2Pr'(a)-2$ strictly decreases for $a\in[0,2)$ as the last two terms decreases in that interval. $Pr'(1)=0$ and thus $Pr'(a)>0,\ \forall a\in[0,1)$. $Pr(a)$ thus strictly increases for $a\in[0,1]$ and $\frac12=Pr(0)\le Pr(a)\le Pr(1)=1$.
  
original_text: 8725: <p>I am reading Section 5.2, Kinship and Inbreeding Coefficients, of <a href="https://books.google.com/books?id=CBzjBwAAQBAJ&amp;pg=PA81&amp;lpg=PA81&amp;dq=if%20they%20are%20both%20physical%20copies%20of%20the%20same%20ancestral%20gene.%20Two%20genes%20are%20identical%20by%20state%20if%20they%20represent%20the%20same%20allele.%20Identity%20by%20descent%20implies%20identity%20by%20state,%20but%20not%20conversely.%20The%20simplest%20measure%20of%20relationship%20between%20two%20relatives&amp;source=bl&amp;ots=730RYtaQkZ&amp;sig=ACfU3U1XOc2uutgH-akYsZINUPwng_AJOQ&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwihr9qGkv7qAhVCnKwKHQNMCuAQ6AEwBHoECAkQAQ#v=onepage&amp;q=if%20they%20are%20both%20physical%20copies%20of%20the%20same%20ancestral%20gene.%20Two%20genes%20are%20identical%20by%20state%20if%20they%20represent%20the%20same%20allele.%20Identity%20by%20descent%20implies%20identity%20by%20state%2C%20but%20not%20conversely.%20The%20simplest%20measure%20of%20relationship%20between%20two%20relatives&amp;f=false" rel="nofollow noreferrer">Kenneth Lange, Mathematical and Statistical Methods for Genetic Analysis</a>. There the kinship coefficient <span class="math-container">$\Phi_{i,j}$</span> is defined for two relatives <span class="math-container">$i$</span> and <span class="math-container">$j$</span> as the probability that a gene selected randomly from <span class="math-container">$i$</span> and a gene selected randomly from the same autosomal locus of <span class="math-container">$j$</span> are identical by descent.</p>
  <p>I would think <span class="math-container">$\Phi_{i,j}$</span> should depend on the particular population frequency of the gene or allele at that locus. However, the book does not seem to indicate that or at least does not stress that at all. Is there a dependence or not? The same question applies to the inbreeding coefficient.</p>
  <p>More importantly, I doubt this definition is mathematically rigorous.</p>
  <hr />
  <p>Here is an example to make what puzzles me clearer. It must be that the probability I am computing is not what the kinship coefficient. But this is the definition means to me. Some expert please elucidate the correct meaning of the definition.</p>
  <p>Consider the simplest pedigree tree of one family of two parents with brothers <span class="math-container">$A$</span> and <span class="math-container">$B$</span>. Allele <span class="math-container">$a$</span> is detected at a particular locus. Let <span class="math-container">$b$</span> be the other allele of the same gene. We ask for the probability of the brother <span class="math-container">$B$</span> having allele <span class="math-container">$a$</span> conditioned on <span class="math-container">$A$</span> having allele <span class="math-container">$a$</span>.</p>
  <p>In the following, to clutter symbols, we abuse the symbols by use <span class="math-container">$a$</span> to denote the population frequency or the unconditional probability of finding allele <span class="math-container">$a$</span> at the locus for the whole population as <span class="math-container">$a$</span>. The same goes for allele <span class="math-container">$b$</span>. <span class="math-container">$a,b\in[0,1]$</span> and <span class="math-container">$a+b=1$</span>. We solve this problem by listing as follows all possible parental gene configurations, the configuration probability (considering the symmetry of paternal and maternal loci), as well as the probability finding allele <span class="math-container">$a$</span> for either <span class="math-container">$A$</span> or <span class="math-container">$B$</span> at the locus.</p>
  <p><span class="math-container">\begin{align}
  \text{parent config}~~~~ &amp; \text{Pr(config)} &amp; \text{Pr($a$|config)} \\
  aa|aa~~~~~~~~~  &amp;  ~~a^4   &amp; 1 \\
  aa|ab~~~~~~~~~  &amp;  4a^3b   &amp; 1 \\
  aa|bb~~~~~~~~~  &amp;  2a^2b^2 &amp; 1 \\
  ab|ab~~~~~~~~~  &amp;  4a^2b^2 &amp; \frac34 \\
  ab|bb~~~~~~~~~  &amp;  4ab^3   &amp; \frac12 \\
  bb|bb~~~~~~~~~  &amp;  ~~~~b^4 &amp; 0 \\
  \end{align}</span></p>
  <p>The joint conditional probability of both <span class="math-container">$A$</span> and <span class="math-container">$B$</span> have allele <span class="math-container">$a$</span> at the locus is Pr(<span class="math-container">$a$</span>|config)<span class="math-container">$^2$</span>. So the required conditional probability is
  <span class="math-container">$$P(r):=\text{Pr}(B\text{ has } a|A\text{ has }a)=\frac{\sum_\text{config} \text{Pr(config)Pr(a|config)}^2}{\sum_\text{config} \text{Pr(config)Pr(a|config)}} = \frac{r^3+4r^2+2r+4r(\frac34)^2+\frac4{2^2}}{r^3+4r^2+2r+4r\frac34+\frac42}, \quad r:=\frac ab\in [0,\infty).$$</span>
  <span class="math-container">$P(r)$</span> depends on <span class="math-container">$r$</span> which depends on the population frequency of the alleles as I have claimed.</p>
  <p>Consider <span class="math-container">$[r^3+4r^2,2r,3r, 2]$</span> as a positive entry vector. <span class="math-container">$P$</span> is a convex combination of <span class="math-container">$v:=\big[1,1,1,\frac34,\frac12\big]$</span>. So <span class="math-container">$P$</span> is between the minimum and maximum of the entries of <span class="math-container">$v$</span>, and
  <span class="math-container">$$\frac12\le P\le 1.$$</span>
  We see the extrema are achieved at
  <span class="math-container">$$P(r)=\begin{cases}
  \frac12, \quad r=0 \\[0.7ex]
  1,\quad r\to\infty
  \end{cases}$$</span></p>
   8725: <p>The definition as set by Kenneth Lange's book is indeed quite vague and thus unrigorous. It is a probability not conditioned on observing any particular allele at a locus thus not on the population frequency of that allele, but conditioned on the following where we focus on one particular locus. It is rigorously defined as follows</p>
  <p>A graph of lineage contains nodes, which we call persons, of a particular locus. Each node, or person, contains exactly two genes. We draw one and only one directed edge (parental relationship) from a gene of a parent person to the gene of a child person if the latter is inherited from the former.  This is a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" rel="nofollow noreferrer">directed acyclic graph</a> (DAG). Each gene can be connected by no more than <span class="math-container">$1$</span> edge directed to (as opposed to away from) it. The two genes in one person cannot be connected by edges. Two persons are said to be connected if there is an edge pointing from a gene of one node to a gene of the other node. Two persons cannot be connected by more than one edge. For a pair of connected persons and their directed edge between them, the edge is equally probable to connect to one gene as the other gene of a person, i.e. the probability is <span class="math-container">$\frac12$</span>. A pair of genes (persons) are said to be connected if there exists an undirected (simple) path from one gene (person) to the other. Two thus connected genes are called identical by descent.</p>
  <p>In this setting, the DAG with respect to the persons is given and deterministic whilst any directed edge in the DAG assigned to two persons are random with independent probability <span class="math-container">$\frac12$</span> with respect to the genes within each person.</p>
  <p>For a pair of persons on a lineage DAG that is connected, the probability of one gene equally probably chosen amongst the two genes of one person being connected, or identical by descent, to one gene equally probably chosen amongst the two genes of the other person is the coefficient of kinship. It can be proven that the probability is <span class="math-container">$\displaystyle\sum_{p\in P} \frac1{2^{n(p)}}$</span> where <span class="math-container">$P$</span> is the set of all distinct (simple) paths connecting one person (<strong>not</strong> gene) to to the other and <span class="math-container">$n(p)$</span> is the number of persons on person connecting path <span class="math-container">$p$</span>.</p>
  <p>As an additional assumption, the genes that are connected have to be the same and thus assume the same allele.</p>
  <hr />
  <p>With regard to the conditional probability example in the question, the coefficient of kinship is not adequate to be used to compute the desired probability. Part of the reason is the question inquires about the existence of an allele rather than picking one gene with 1/2 of the probability. We need some more fundamental probability to compute that. We will demonstrate that as follows.</p>
  <ol>
  <li>We will compute the probability of the two siblings having both allele <span class="math-container">$a$</span> as follows.</li>
  </ol>
  <p>For a pair of full siblings, they can have exactly <span class="math-container">$0, 1, 2$</span> identical by descent genes. Let the respective probabilities be <span class="math-container">$r_0, r_1, r_2$</span>. Notice we are not picking genes of a person at random but talking about the existence of particular types of genes in a person. For two full siblings, some thoughts lead to <span class="math-container">$r_0=\frac14, r_1=\frac12, r_2=\frac14$</span>.</p>
  <p>By symmetry and without the loss of generality, we fix the connecting genes in the two siblings of the respective edges. We only need to consider the connections of the edges relative to the genes of the parents.</p>
  <p>(1) Conditioned on there is no identical by descent genes, the sibling genes can independently assume any allele. The probability of having the configuration of <span class="math-container">$ab|ab$</span> (not considering gene ordering) is <span class="math-container">$2^2a^2b^2$</span> considering the distinct ordering of the alleles, that of <span class="math-container">$aa|ab$</span> is <span class="math-container">$\binom41 a^3b$</span>, and that of <span class="math-container">$aa|aa$</span> is <span class="math-container">$a^4$</span>. The total probability is thus
  <span class="math-container">$$Pr_0:=\frac14(4a^2b^2+4a^3b+a^4).$$</span></p>
  <p>(2) When there is exactly <span class="math-container">$1$</span> pair of identical by descent genes, the two identical by descent genes can only assume the same allele, while the other genes can assume alleles independently. When the identical by descent genes assume allele <span class="math-container">$a$</span> the probability of the two siblings both having allele <span class="math-container">$a$</span> is <span class="math-container">$a$</span>. When the two identical by descent genes assume allele <span class="math-container">$b$</span> the desired probability is then <span class="math-container">$ba^2$</span>. The total probability is then
  <span class="math-container">$$Pr_1:=\frac12(a+ba^2).$$</span></p>
  <p>(3) When there are exactly <span class="math-container">$2$</span> pairs of identical by descent genes, each pair can only assume the same allele but independently of each other. The only way that the two siblings not to both have allele <span class="math-container">$a$</span> is for the two identical by descent pairs to assume allele <span class="math-container">$b$</span>. The total desired probability for the two siblings to both have allele <span class="math-container">$a$</span> is then
  <span class="math-container">$$Pr_2:=\frac14(1-b^2)$$</span></p>
  <ol start="2">
  <li>Now we compute the probability of any one of the siblings having allele <span class="math-container">$a$</span>. Each gene can of course assume alleles independently. So the probability is simply
  <span class="math-container">$$Pr_3:=1-b^2.$$</span></li>
  </ol>
  <p>Therefor the desired probability of both siblings having allele <span class="math-container">$a$</span> given that one of them has allele <span class="math-container">$a$</span> is
  <span class="math-container">$$Pr(a):=\frac{Pr_0+Pr_1+Pr_2}{Pr_3}=\frac{\frac14a(a^3-6a^2+5a+4)}{a(2-a)}$$</span>
  since <span class="math-container">$b=1-a$</span>. Some tedious algebra which can be carried out by, say, Mathematica shows this is identical to the expression in the question in terms of <span class="math-container">$r=\frac{a}{1-a}$</span>.
  <span class="math-container">$$Pr'(a)=\frac12\Big(2-\frac1{(2-a)^2}-a\Big).$$</span>
  It is easy to see <span class="math-container">$2Pr'(a)-2$</span> strictly decreases for <span class="math-container">$a\in[0,2)$</span> as the last two terms decreases in that interval. <span class="math-container">$Pr'(1)=0$</span> and thus <span class="math-container">$Pr'(a)&gt;0,\ \forall a\in[0,1)$</span>. <span class="math-container">$Pr(a)$</span> thus strictly increases for <span class="math-container">$a\in[0,1]$</span> and <span class="math-container">$\frac12=Pr(0)\le Pr(a)\le Pr(1)=1$</span>.</p>
  