A method, computer program product, and computer system, for receiving a first set of ground truth instances from a first source. A second set of ground truth instances may be received from a second source. The first and second sets of ground truth instances may be weighed differently based on a level of trust associated with each of the first and second sources. The weighted first and second sets of ground truth instances may be applied in a machine learning task executed by a computer.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A computer-implemented method comprising: receiving, by a computing device, a first set of ground truth instances from a first source; receiving a second set of ground truth instances from a second source; weighing the first and second sets of ground truth instances differently based on a level of trust associated with each of the first and second sources; and applying the weighted first and second sets of ground truth instances in a machine learning task executed by a computer. 2. The computer-implemented method of claim 1 wherein applying the weighted first and second sets of ground truth instances in the machine learning task includes adjusting a loss function for at least one of the first and second sets of ground truth instances based upon, at least in part, the level of trust associated with each of the first and second sources. 3. The computer-implemented method of claim 2 wherein applying the weighted first and second sets of ground truth instances in the machine learning task includes at least one of a mean-squared error and duplication of instances for the first set of ground truth instances more than the second set of ground truth instances. 4. The computer-implemented method of claim 1 further comprising: receiving a third set of ground truth instances from a third source; and weighing the third set of ground truth instances differently than the first and second sets of ground truth instances based on a level of trust associated with each of the first, second, and third sources, wherein the weighted first, second, and third sets of ground truth instances are applied in the machine learning task. 5. The computer-implemented method of claim 1 wherein the level of trust associated with the first set of ground truth instances is indicative of validation of the first set of ground truth instances by a subject matter expert. 6. The computer-implemented method of claim 1 wherein the level of trust associated with the second set of ground truth instances is indicative of validation of the second set of ground truth instances by QA system developer, and wherein the level of trust associated with the third set of ground truth instances is indicative of validation of the third set of ground truth instances by a threshold number of crowdsource users. 7. The computer-implemented method of claim 1 further comprising updating an answer in the second set of ground truth instances for association with the first set of ground truth instances in response to receiving validation of the answer from a subject matter expert. 8.-20. (canceled)