A method for training a question answering system includes providing training questions to a question answering system executing on a computer and to a plurality of subject matter experts. The question answering system generates first answers to each training question. Second answers to the training questions are received from the subject matter experts. Feature scores for each of the first answers and the second answers are generated and compared across the second answers and the first answers. Each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question. Based on the comparing, a measure of consistency of the feature scores of the second answers is determined, and a measure of consistency of the feature scores of the second answers to the first answers is determined. The measures of consistency are transmitted to the subject matter experts.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method comprising: providing training questions to a question answering system executing on a computer; producing, by the question answering system, first answers to each of the training questions; providing the training questions to a plurality of subject matter experts; receiving, from the subject matter experts, second answers to the training questions; generating a plurality of feature scores for each of the first answers and the second answers, wherein each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question; for each of the training questions, comparing the feature scores across the second answers and the first answers; determining, based on results of the comparing: a measure of consistency with respect to the feature scores of the second answers to one another; and a measure of consistency with respect to the feature scores of the second answers to the first answers; and transmitting results of the determining to at least one of the subject matter experts. 2. The method of claim 1, wherein the generating of feature scores comprises evaluating the first answers and the second answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group comprising question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers. 3. The method of claim 2, further comprising identifying which of the scoring features are associated with features scores of the second answers that exceed a predetermined threshold across a predetermined number of the subject matter experts. 4. The method of claim 3, further comprising configuring the question answering system to rank potential answers to a question based on feature scores corresponding to the scoring features associated with features scores of the second answers that exceed the predetermined threshold. 5. The method of claim 1, further comprising identifying, based on results of the comparing, at least one of the subject matter experts corresponding to feature scores that exhibit less than a predetermined degree of similarity to the feature scores corresponding to others of the subject matter experts. 6. The method of claim 5, wherein the transmitting comprises providing the results of determining to the at least one of the subject matter experts. 7. The method of claim 1, wherein the comparing comprises: generating a vector indicative of value of each of the feature scores in each of the first answers and the second answers; and determining a distance between each two of the vectors. 8. A system comprising: a question answering system executed by a computer; a processor; and a memory coupled to the processor, the memory encoded with instructions that when executed cause the processor to provide a training system for training the question answering system, the training system configured to: provide training questions to the question answering system; retrieve, from the question answering system first answers to each of the training questions; provide the training questions to a plurality of subject matter experts; receive, from the subject matter experts, second answers to the training questions; generate a plurality of feature scores for each of the first answers and the second answers, wherein each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question; for each of the training questions, compare the feature scores across the second answers and the first answers; determine, based on results of comparing the feature scores across the second answers and the first answers: a first measure of consistency with respect to the feature scores of the second answers to one another; and a second measure of consistency with respect to the feature scores of the second answers to the first answers; and transmit information indicative of at least one of the first measure of consistency and the second measure of consistency to one or more of the subject matter experts. 9. The system of claim 8, wherein the training system is configured to evaluate the first answers and the second answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group comprising question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers. 10. The system of claim 9, wherein the training system is configured to identify which of the scoring features are associated with features scores of the second answers that exceed a predetermined threshold across a predetermined number of the subject matter experts. 11. The system of claim 10, wherein the training system is configured to configure the question answering system to rank potential answers to a question based on feature scores corresponding to the scoring features associated with features scores of the second answers that exceed the predetermined threshold. 12. The system of claim 8, wherein the training system is configured to identify, based on the first measure of consistency, at least one of the subject matter experts corresponding to feature scores that exhibit less than a predetermined degree of similarity to the feature scores corresponding to others of the subject matter experts. 13. The system of claim 12, wherein the training system is configured to transmit the information to the at least one of the subject matter experts. 14. The system of claim 8, wherein the training system is configured to: generate a vector indicative of value of each of the feature scores in each of the first answers and the second answers; and determine a distance between each two of the vectors. 15. A computer program product for training a question answering system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to: provide training questions to the question answering system; retrieve, from the question answering system first answers to each of the training questions; provide the training questions to a plurality of subject matter experts; receive, from the subject matter experts, second answers to the training questions; generate a plurality of feature scores for each of the first answers and the second answers, wherein each of the feature scores is representative of a quality of an answer that is indicative of relevance to a corresponding training question; for each of the training questions, compare the feature scores across the second answers and the first answers; determine, based on results of comparing the feature scores across the second answers and the first answers: a first measure of consistency with respect to the feature scores of the second answers to one another; and a second measure of consistency with respect to the feature scores of the second answers to the first answers; and transmit information indicative of at least one of the first measure of consistency and the second measure of consistency to one or more of the subject matter experts. 16. The computer program product of claim 15, wherein the program instructions are executable by the computer to cause the computer to evaluate the first answers and the second answers with respect to a plurality of scoring features comprising one or more scoring features selected from a group comprising question type, focus, lexical answer type, sentence structure, term matching, and grammatical modifiers. 17. The computer program product of claim 16, wherein the program instructions are executable by the computer to cause the computer to identify which of the scoring features are associated with features scores of the second answers that exceed a predetermined threshold across a predetermined number of the subject matter experts. 18. The computer program product of claim 17, wherein the program instructions are executable by the computer to cause the computer to configure the question answering system to rank potential answers to a question based on feature scores corresponding to the scoring features associated with features scores of the second answers that exceed the predetermined threshold. 19. The computer program product of claim 15, wherein the program instructions are executable by the computer to cause the computer to identify, based on the first measure of consistency, at least one of the subject matter experts corresponding to feature scores that exhibit less than a predetermined degree of similarity to the feature scores corresponding to others of the subject matter experts. 20. The computer program product of claim 19, wherein the program instructions are executable by the computer to cause the computer to transmit the information to the at least one of the subject matter experts.