Reading between the lines, it may be that the child's biological father and your husband's own offspring-situation are part of the equation.
Your child's birth father and your husband's children could be part of the problem.
Your father's children are the only part of the problem.