An active learning framework is operative to identify informative questions that should be added to existing question-answer (Q&A) pairs that comprise a training dataset for a learning model. In this approach, the question—answer pairs (to be labeled as “true” or “false”) are automatically selected from a larger pool of unlabeled data. A spatial-directed clustering algorithm partitions the relevant question-answer space of unlabeled data. A margin-induced loss function is then used to rank a question. For each question selected, a label is then obtained, preferably by assigning a prediction for each associated question-answer pair using a current model that has been trained on labeled question-answer pairs. After the questions are labeled, an additional re-sampling is performed to assure high quality of the training data. Preferably, and with respect to a particular question, this additional re-sampling is based on a distance measure between correct and incorrect answers.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method to identify questions for machine learning, comprising: training an initial machine learning statistical model using a set of labeled question-answer pairs; receiving and evaluating a set of unlabeled data that includes question-answer pairs; ranking questions derived from the set of unlabeled data in part using a question loss function that represents a loss value computed for a particular question having a set of one or more answers, wherein the loss value for the particular question is computed as a function of loss values for the one or more answers associated therewith; selecting questions from the ranking to produce a set of discriminative questions; and applying labels to the set of discriminative questions using the initial machine learning model; wherein at least one or more operations are performed in software executing in a hardware element. 2. The method as described in claim 1 further including using the set of discriminative questions to train a new machine learning statistical model. 3. The method as described in claim 1 wherein evaluating the unlabeled data includes pre-processing question-answer pairs by partitioning an answer space into a spatial-oriented data structure. 4. The method as described in claim 3 wherein the pre-processing uses a spatial-directed clustering algorithm. 5. The method as described in claim 3 further including re-sampling a subset of the questions selected from the ranking by re-sampling data from the spatial-oriented data structure. 6. The method as described in claim 5 wherein the re-sampling is based on a distance measure between correct and incorrect answers in the unlabeled data. 7. The method as described in claim 1 wherein the question loss function is computed by: computing a loss of each answer; and computing an average loss of the answers for a question; wherein questions are ranked according to an algorithm that minimizes a selected number of questions from the set of questions and reduces the average loss for at least one question. 8. Apparatus, comprising: a processor; and computer memory holding computer program instructions executed by the processor, to identify questions for machine learning, the computer program instructions operative to: train an initial machine learning statistical model using a set of labeled question-answer pairs; receive and evaluate a set of unlabeled data that includes question-answer pairs; rank questions derived from the set of unlabeled data in part using a question loss function that represents a loss value computed for a particular question having a set of one or more answers, wherein the loss value for the particular question is computed as a function of loss values for the one or more answers associated therewith; select questions from the ranking to produce a set of discriminative questions; and apply labels to the set of discriminative questions using the initial machine learning model. 9. The apparatus as described in claim 8 wherein the computer program instructions are further operative to use the set of discriminative questions to train a new machine learning statistical model. 10. The apparatus as described in claim 8 wherein the computer program instructions are further operative to pre-process question-answer pairs by partitioning an answer space into a spatial-oriented data structure. 11. The apparatus as described in claim 10 wherein the pre-processing uses a spatial-directed clustering algorithm. 12. The apparatus as described in claim 10 wherein the computer program instructions are further operative to re-sample a subset of the questions selected from the ranking by re-sampling data from the spatial-oriented data structure. 13. The apparatus as described in claim 12 wherein the re-sampling is based on a distance measure between correct and incorrect answers in the unlabeled data. 14. The apparatus as described in claim 8 wherein the loss function is computed by computer program instructions that are further operative to: compute a loss of each answer; and compute an average loss of the answers for a question; wherein questions are ranked according to an algorithm that minimizes a selected number of questions from the set of questions and reduces the average loss for at least one question. 15. A computer program product in a non-transitory computer readable storage medium for use in a computing entity, the computer program product holding computer program instructions which, when executed, identify questions for machine learning, the computer program instructions operative to: train an initial machine learning statistical model using a set of labeled question-answer pairs; receive and evaluate a set of unlabeled data that includes question-answer pairs; rank questions derived from the set of unlabeled data in part using a question loss function that represents a loss value computed for a particular question having a set of one or more answers, wherein the loss value for the particular question is computed as a function of loss values for the one or more answers associated therewith; select questions from the ranking to produce a set of discriminative questions; and apply labels to the set of discriminative questions using the initial machine learning model. 16. The computer program product as described in claim 15 wherein the computer program instructions are further operative to use the set of discriminative questions to train a new machine learning statistical model. 17. The computer program product as described in claim 15 wherein the computer program instructions are further operative to pre-process question-answer pairs by partitioning an answer space into a spatial-oriented data structure. 18. The computer program product as described in claim 17 wherein the pre-processing uses a spatial-directed clustering algorithm. 19. The computer program product as described in claim 15 wherein the computer program instructions are further operative to re-sample a subset of the questions selected from the ranking by re-sampling data from the spatial-oriented data structure. 20. The computer program product as described in claim 19 wherein the re-sampling is based on a distance measure between correct and incorrect answers in the unlabeled data. 21. The computer program product as described in claim 15 wherein the question loss function is computed by computer program instructions that are further operative to: compute a loss of each answer; and compute an average loss of the answers for a question; wherein questions are ranked according to an algorithm that minimizes a selected number of questions from the set of questions and reduces the average loss for at least one question.