A method for ground truth generation includes providing training questions to a machine learning system executing on a computer. The machine learning system generates candidate answers to each of the training questions. The method also includes providing the candidate answers to a plurality of subject matter experts for evaluation with respect to the training questions, wherein the evaluation comprises assignment of an SME relevance score to each of the candidate answers. The method further includes analyzing each of the candidate answers with respect to a plurality of scoring features, wherein each of the scoring features is indicative of quality of the candidate answer. The method yet further includes generating a ground truth metric value that indicates a measure of agreement between the subject matter experts relative to a measure of agreement between results of the analyzing.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method comprising: providing training questions to a machine learning system executing on a computer; producing, by the machine learning system: candidate answers to each of the training questions; and a system relevance score for each of the candidate answers; providing the candidate answers to a plurality of subject matter experts for evaluation with respect to the training questions, wherein the evaluation comprises assignment of an SME relevance score to each of the candidate answers; analyzing each of the candidate answers with respect to a plurality of scoring features, wherein each of the scoring features is indicative of quality of the candidate answer; and generating a ground truth metric value that indicates a measure of agreement between the subject matter experts relative to a measure of agreement between results of the analyzing. 2. The method of claim 1, further comprising providing the ground truth metric to the subject matter experts, wherein the ground truth metric guides the subject matter experts as to quality of the evaluation of the candidate answers. 3. The method of claim 1, wherein the analyzing comprises, for each of the scoring features, generating a vector indicative of value of each of the scoring features in determining the quality of the candidate answer. 4. The method of claim 3, wherein the analyzing comprises determining a distance between each two of the vectors. 5. The method of claim 4, wherein the generating comprises producing the ground truth metric as a ratio of average difference of subject matter expert vectors to average difference of subject matter expert vectors and candidate answer vectors. 6. The method of claim 5, further comprising excluding from the generating of the ground truth metric each subject matter expert vector that differ from a candidate answer vector by less than a predetermined amount. 7. The method of claim 1, further comprising normalizing the system relevance scores to the SME relevance scores. 8. The method of claim 1, wherein value of the ground truth metric increases with agreement between the subject matter experts and decreases with agreement between the subject matter experts and the candidate answers. 9. A system comprising: a machine learning system executed by a computer; a processor; and a memory coupled to the processor, the memory encoded with instructions that when executed cause the processor to provide a training system for training the machine learning system, the training system configured to: provide training questions to the machine learning system; retrieve, from the machine learning system: candidate answers to each of the training questions; and a system relevance score for each of the candidate answers; provide the candidate answers to a plurality of subject matter experts for evaluation with respect to the training questions, wherein the evaluation comprises assignment of an SME relevance score to each of the candidate answers; analyze each of the candidate answers with respect to a plurality of scoring features, wherein each of the scoring features is indicative of quality of the candidate answer; and generate a ground truth metric value that indicates a measure of agreement between the subject matter experts relative to a measure of agreement between results of the analyzing. 10. The system of claim 9, wherein the training system is configured to provide the ground truth metric to the subject matter experts, wherein the ground truth metric guides the subject matter experts as to quality of the evaluation of the candidate answers. 11. The system of claim 9, wherein the training system is configured to: for each of the scoring features, generate a vector indicative of value of each of the scoring features in determining the quality of the candidate answer; and determine a distance between each two of the vectors. 12. The system of claim 11, wherein the training system is configured to generate the ground truth metric as a ratio of average difference of subject matter expert vectors to average difference of subject matter expert vectors and candidate answer vectors. 13. The system of claim 12, wherein the training system is configured to exclude from generation of the ground truth metric each subject matter expert vector that differs from a candidate answer vector by less than a predetermined amount. 14. The system of claim 9, wherein the training system is configured to normalize the system relevance scores to the SME relevance scores. 15. The system of claim 9, wherein value of the ground truth metric increases with agreement between the subject matter experts and decreases with agreement between the subject matter experts and the candidate answers. 16. A computer program product for training a machine learning system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to: provide training questions to the machine learning system; retrieve, from the machine learning system: candidate answers to each of the training questions; and a system relevance score for each of the candidate answers; provide the candidate answers to a plurality of subject matter experts for evaluation with respect to the training questions, wherein the evaluation comprises assignment of an SME relevance score to each of the candidate answers; analyze each of the candidate answers with respect to a plurality of scoring features, wherein each of the scoring features is indicative of quality of the candidate answer; generate a ground truth metric value that indicates a measure of agreement between the subject matter experts relative to a measure of agreement between results of the analyzing. 17. The computer program product of claim 16, wherein the program instructions are executable by the computer to cause the computer to provide the ground truth metric to the subject matter experts, wherein the ground truth metric guides the subject matter experts with information indicative of quality of the evaluation. 18. The computer program product of claim 16, wherein the program instructions are executable by the computer to cause the computer to: normalize the system relevance scores to the SME relevance scores; for each of the scoring features, generate a vector indicative of value of each of the scoring features in determining the quality of the candidate answer; and determine a distance between each two of the vectors. 19. The computer program product of claim 18, wherein the program instructions are executable by the computer to cause the computer to generate the ground truth metric as a ratio of average difference of subject matter expert vectors to average difference of subject matter expert vectors and candidate answer vectors. 20. The computer program product of claim 19, wherein the program instructions are executable by the computer to cause the computer to exclude from generation of the ground truth metric each subject matter expert vector that differs from a candidate answer vector by less than a predetermined amount.