Complete a TPED for duplicated SNPs.

    :param tped: a representation of the ``tped`` of duplicated markers.
    :param tfam: a representation of the ``tfam``.
    :param snps: the position of duplicated markers in the ``tped``.
    :param prefix: the prefix of the output files.
    :param chosenSNPs: the markers that were chosen for completion (including
                       problems).
    :param completion: the completion of each of the duplicated markers.
    :param concordance: the pairwise concordance of the duplicated markers.
    :param snpsToComplete: the markers that will be completed (excluding
                           problems).
    :param tfamFileName: the name of the original ``tfam`` file.
    :param completionT: the completion threshold.
    :param concordanceT: the concordance threshold.

    :type tped: numpy.array
    :type tfam: list
    :type snps: dict
    :type prefix: str
    :type chosenSNPs: dict
    :type completion: numpy.array
    :type concordance: dict
    :type snpsToComplete: set
    :type tfamFileName: str
    :type completionT: float
    :type concordanceT: float

    :returns: a tuple containing the new ``tped`` after completion
              (:py:class:`numpy.array` as the first element, and the index of
              the markers that will need to be rid of (:py:class:`set`) as the
              last element.

    It creates three different files:

    * ``prefix.zeroed_out``: contains information about markers and samples
                             where the genotyped was zeroed out.
    * ``prefix.not_good_enough``: contains information about markers that were
                                  not good enough to help in completing the
                                  chosen markers (because of concordance or
                                  completion).
    * ``prefix.removed_duplicates``: the list of markers that where used for
                                     completing the chosen one, hence they will
                                     be removed from the final data
                                     set.

    Cycling through every genotypes of every samples of every duplicated
    markers, checks if the genotypes are all the same. If the chosen one was
    not called, but the other ones were, then we complete the chosen one with
    the genotypes for the others (assuming that they are all the same). If
    there is a difference between the genotypes, it is zeroed out for the
    chosen marker.

def createAndCleanTPED(tped, tfam, snps, prefix, chosenSNPs, completion,
                       concordance, snpsToComplete, tfamFileName, completionT,
                       concordanceT):
    """Complete a TPED for duplicated SNPs.

    :param tped: a representation of the ``tped`` of duplicated markers.
    :param tfam: a representation of the ``tfam``.
    :param snps: the position of duplicated markers in the ``tped``.
    :param prefix: the prefix of the output files.
    :param chosenSNPs: the markers that were chosen for completion (including
                       problems).
    :param completion: the completion of each of the duplicated markers.
    :param concordance: the pairwise concordance of the duplicated markers.
    :param snpsToComplete: the markers that will be completed (excluding
                           problems).
    :param tfamFileName: the name of the original ``tfam`` file.
    :param completionT: the completion threshold.
    :param concordanceT: the concordance threshold.

    :type tped: numpy.array
    :type tfam: list
    :type snps: dict
    :type prefix: str
    :type chosenSNPs: dict
    :type completion: numpy.array
    :type concordance: dict
    :type snpsToComplete: set
    :type tfamFileName: str
    :type completionT: float
    :type concordanceT: float

    :returns: a tuple containing the new ``tped`` after completion
              (:py:class:`numpy.array` as the first element, and the index of
              the markers that will need to be rid of (:py:class:`set`) as the
              last element.

    It creates three different files:

    * ``prefix.zeroed_out``: contains information about markers and samples
                             where the genotyped was zeroed out.
    * ``prefix.not_good_enough``: contains information about markers that were
                                  not good enough to help in completing the
                                  chosen markers (because of concordance or
                                  completion).
    * ``prefix.removed_duplicates``: the list of markers that where used for
                                     completing the chosen one, hence they will
                                     be removed from the final data
                                     set.

    Cycling through every genotypes of every samples of every duplicated
    markers, checks if the genotypes are all the same. If the chosen one was
    not called, but the other ones were, then we complete the chosen one with
    the genotypes for the others (assuming that they are all the same). If
    there is a difference between the genotypes, it is zeroed out for the
    chosen marker.

    """
    zeroedOutFile = None
    try:
        zeroedOutFile = open(prefix + ".zeroed_out", "w")
    except IOError:
        msg = "%(prefix).zeroed_out: can't write file" % locals()
        raise ProgramError(msg)
    print >>zeroedOutFile, "\t".join(["famID", "indID", "snpID"])

    notGoodEnoughFile = None
    try:
        notGoodEnoughFile = open(prefix + ".not_good_enough", "w")
    except IOError:
        msg = "%(prefix)s.not_good_enough: can't write file" % locals()
        raise ProgramError(msg)
    print >>notGoodEnoughFile, "\t".join(["name", "reason"])

    removedFile = None
    try:
        removedFile = open(prefix + ".removed_duplicates", "w")
    except IOError:
        msg = "%(prefix)s.removed_duplicates: can't write file" % locals()
        raise ProgramError(msg)

    notGoodEnoughSnps = set()

    # Split the tped in 'snpInfo' and 'genotypes'
    snpInfo = tped[:, :4]
    genotypes = tped[:, 4:]

    # The sed of index we want to get rid of at the end
    getRidOfIndex = set()

    for snpID, indexes in snps.iteritems():
        if snpID not in snpsToComplete:
            # We don't want to complete this SNP, so we continue to next SNP
            continue

        # Getting the completion
        completionToRemove = set(
            np.where(completion[indexes] < completionT)[0]
        )
        for k in completionToRemove:
            notGoodEnoughSnps.add((snpInfo[indexes][k, 1], "completion"))

        # Getting the concordance
        concordanceToRemove = set(
            np.where(concordance[snpID] < concordanceT)[0]
        )
        for k in concordanceToRemove:
            notGoodEnoughSnps.add((snpInfo[indexes][k, 1], "concordance"))

        # These will be the indexes to remove
        indexesToRemove = set()
        for index in completionToRemove | concordanceToRemove:
            indexesToRemove.add(indexes[index])

        # These are the indexes to keep
        indexesToKeep = []
        for index in indexes:
            if index not in indexesToRemove:
                indexesToKeep.append(index)

        # Getting the chosen SNP
        chosenOne = chosenSNPs[snpID]
        if chosenOne not in set(indexesToKeep):
            # The chosen SNP is not a good SNP, so we go to next SNP
            logger.warning("  - {} chosen but not good enough".format(
                snpInfo[chosenOne, 1],
            ))
            continue

        # Now cycling through the genotypes
        nbSamples = genotypes.shape[1]
        for sampleIndex in xrange(nbSamples):
            # We need to remove the no call and keep the unique genotypes
            curGenotypes = genotypes[indexesToKeep, sampleIndex]
            cleanedCurGenotypes = curGenotypes[
                np.where(curGenotypes != "0 0")
            ]
            uniqueCleanedCurGenotypes = np.unique(cleanedCurGenotypes)

            # Checking the number of unique genotypes
            toComplete = False
            if len(uniqueCleanedCurGenotypes) > 1:
                # There are more than one unique genotype (except 0 0)
                # len = 0 means all were 0 0
                # len = 1 means they are all the same
                # len > 1 means discordance (might need to flip)
                # Just need to check the order of the alleles
                possibleAlleles = [
                    set() for k in xrange(len(uniqueCleanedCurGenotypes))
                ]
                for k, geno in enumerate(uniqueCleanedCurGenotypes):
                    possibleAlleles[k] |= set(geno.split(" "))
                allEqual = True
                for k in xrange(len(possibleAlleles)):
                    for l in xrange(k+1, len(possibleAlleles)):
                        if possibleAlleles[k] != possibleAlleles[l]:
                            allEqual = False

                if not allEqual:
                    # The genotypes are not all equal, we set the chosen
                    # genotype to null (0 0)
                    tped[chosenOne, sampleIndex+4] = "0 0"
                    print >>zeroedOutFile, "\t".join([tfam[sampleIndex, 0],
                                                      tfam[sampleIndex, 1],
                                                      snpInfo[chosenOne, 1]])
                elif genotypes[chosenOne, sampleIndex] == "0 0":
                    toComplete = True
            elif ((len(uniqueCleanedCurGenotypes) == 1) and
                    (genotypes[chosenOne, sampleIndex] == "0 0")):
                toComplete = True

            if toComplete:
                # We complete the current individual
                tped[chosenOne, sampleIndex+4] = uniqueCleanedCurGenotypes[0]

        # We keep only the chose one
        for index in indexes:
            if index != chosenOne:
                getRidOfIndex.add(index)
                print >>removedFile, snpInfo[index, 1]

    # Writing the not good enough file
    for item in notGoodEnoughSnps:
        print >>notGoodEnoughFile, "\t".join(item)

    # Closing the output files
    zeroedOutFile.close()
    notGoodEnoughFile.close()

    # Printing the chosen file
    try:
        shutil.copy(tfamFileName, prefix + ".chosen_snps.tfam")
    except IOError:
        msg = "%(tfamFileName)s: can't copy file to " \
              "%(prefix)s.chosen_snps.tfam" % locals()
        raise ProgramError(msg)
    chosenFile = None
    try:
        chosenFile = open(prefix + ".chosen_snps.tped", "w")
    except IOError:
        msg = "%(prefix)s.chosen_snps.tped: can't write file" % locals()
        raise ProgramError(msg)
    for chosenOne in chosenSNPs.itervalues():
        snpID = (tped[chosenOne, 0], tped[chosenOne, 3])
        if snpID in snpsToComplete:
            print >>chosenFile, "\t".join(tped[chosenOne])
    chosenFile.close()

    return tped, getRidOfIndex