The present disclosure relates to the electronic document review field and, more particularly, to various apparatuses and methods of implementing batch-mode active learning for technology-assisted review (TAR) of documents (e.g., legal documents).
Please help me write a proper abstract based on the patent claims. CLAIM: 1. An apparatus configured to implement batch-mode active learning for technology-assisted review (TAR) of documents, the apparatus comprising: a processor; and, a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the apparatus is operable to: obtain an unlabeled set of documents D; obtain a batch size k; construct a first batch of k documents D; obtain labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; construct a classification model, Mc, using the labeled first batch of k documents D; perform an iteration of active learning using the classification model Mc, wherein the perform operation comprises: (i) select a new batch of unlabeled instances Bc using a current version of the classification model, Mc(x), an unlabeled set of available documents D, and the batch size k, wherein the apparatus is further operable to implement a Diversity Sampler process or a Biased Probabilistic Sampler process to select the new batch of unlabeled instances Bc; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; construct an updated classification model M(x) using the extended training data documents Dc; determine whether a stopping criteria has been met; based on the determination that the stopping criteria has not been met, repeat the perform operation, the third construct operation, and the determine operation; and, based on the determination that the stopping criteria has been met, return the updated classification model M(x). 2. The apparatus of claim 1, wherein the classification model, Mc, comprises one of the following: a support vector machine model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model, or ensemble model. 3. The apparatus of claim 1, wherein the apparatus is further operable to implement the Diversity Sampler process as follows: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), and a similarity threshold t; sort the unlabeled set of available documents D using a sorting function to obtain sorted indices I of the unlabeled set of available documents D; insert the sorted document having a nearest sorted indice I[1] of the sorted unlabeled set of available documents D into the new batch of unlabeled instances Bc; obtain sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; remove the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and repeat the insert operation, the second obtain operation, and the remove operation until insert k documents into the new batch of unlabeled instances Bc. 4. The apparatus of claim 3, wherein the sorting function is one of the following: a distance from a hyperplane based on smallest to greatest when the classification model Mc(x) is a support vector machine, SVM, classification model; a distance from a hyperplane based on greatest to smallest when the classification model Mc(x) is a support vector machine, SVM, classification model; a random sorting function; an entropy function; a least confidence function; or a committee disagreement function. 5. The apparatus of claim 1, wherein the apparatus is further operable to implement the Biased Probabilistic Sampler process as follows: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), a similarity threshold t, and a probability distribution function P(x); construct a probability vector w based on an probability distribution function P(x) for each document from the unlabeled set of available documents D; choose a document I[1] from the unlabeled set of available documents D using the weight w of the probability vector; insert the chosen document I[1] into the new batch of unlabeled instances Bc; obtain sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; remove the documents with the obtained sorted indices I from the unlabeled set of available documents D; re-normalize the weight vector as documents have been removed from the sorted unlabeled set of available documents D; and, repeat the choose operation, the insert operation, the second obtain operation, the remove operation, and the re-normalize operation until select the new batch of unlabeled instances Bc. 6. The apparatus of claim 1, wherein the apparatus is further operable to determine whether the stopping criteria has been met by implementing a Kappa agreement which is a measure of how much the current version of the hyperplane hc(x) and the updated hyperplane h(x) agree on respective prediction of labels of a chosen set of the documents D. 7. The apparatus of claim 1, wherein the apparatus is further operable to determine whether the stopping criteria has been met by implementing an Accumulated History of Scores process which comprises the following operations: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, an accumulation function A(Sc, Sc-1, . . . , S1), and a stopping threshold tstop; construct a score vector Sc using the current classification model Mc(x) and the total set of documents DT; combine a current score vector Sc with previous score vectors (Sc-1, . . . , S1) using the accumulation function A(Sc, Sc-1, . . . , S1) to obtain a stability value (s); compare the stability value s to the stopping threshold tstop to determine whether tstop≦S which indicates that the stopping criteria has been met; and based on the determination that the stopping criteria has not been met, store the current score vector Sc to memory as a previous score vector Sc-1. 8. The apparatus of claim 1, wherein the apparatus is further operable to determine whether the stopping criteria has been met by implementing a Cohen's Kappa process which comprises the following operations: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, and a stopping threshold tstop; construct a score vector Sc using the current classification model Mc(x) and the total set of documents DT; retrieve a previous score vector Sc-i; obtain a set of documents D+c from the documents DT that have a positive score in Sc; obtain a set of documents D−c from the documents DT that have a negative score in Sc obtain a set of documents D+(c-1) from the documents DT that have a positive score in Sc-1; obtain a set of documents D−(c-1) from the documents DT that have a negative score in Sc-1; obtain a set of documents in common, D+, between D+c and D+(c-1); obtain a set of documents in common, D−, between D−c and D−(c-1); obtain a probability, P+, of a document having positive score in both score vectors Sc and Sc-1 by counting a number of documents, N+, in D+ divided by the total number of documents, N, in D; obtain a probability, P−, of a document having negative score in both score vectors Sc and Sc-1, by counting a number of documents, N−, in D− divided by the total number of documents, N, in D; obtain a value Ao as P++P−; obtain a probability, P+c, by counting a number of documents, N+c, in D+c divided by the number of documents, N, in D; obtain a probability, P−c, by counting a number of documents, N−c, in D−c divided by the number of documents, N, in D; obtain a probability, P+(c-1), by counting a number of documents, N+(c-1), in D−(c-1) divided by the number of documents, N, in D; obtain a probability, P−(c-1), by counting a number of documents, N−(c-1), in D−(c-1) divided by the number of documents, N, in D; obtain a value Ae as a probability of obtaining a positive document, P+c*P+(c-1), plus a probability of obtaining a negative document, P−c*P−(c-1); obtain a Kappa value L as Ao−Ae divided by (1−Ao); compare the Kappa value L to the stopping threshold tstop to determine whether tstop≦Ao−Ae/1−Ae which indicates that the stopping criteria has been met; and, based on the determination that the stopping criteria has not been met, store the score vector Sc to memory as a previous score vector Sc-1. 9. The apparatus of claim 1, wherein the apparatus is further operable to: discriminate, using the updated hyperplane h(x), remaining documents from the unlabeled set of available documents D where each of the remaining documents are classified as relevant or non-relevant. 10. A method in an apparatus for implementing batch-mode active learning for technology-assisted review (TAR) of documents, the method comprising: obtaining an unlabeled set of documents D; obtaining a batch size k; constructing a first batch of k documents D; obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; constructing a classification model, Mc, using the labeled first batch of k documents D; performing an iteration of active learning using the classification model Mc, wherein the perform operation comprises: (i) selecting a new batch of unlabeled instances Bc using a current version of the classification model, Mc(x), an unlabeled set of available documents D, and the batch size k, wherein the apparatus is further operable to implement a Diversity Sampler process or a Biased Probabilistic Sampler process to select the new batch of unlabeled instances Bc; (ii) obtaining labels for the new batch of unlabeled instances Bc; (iii) adding the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; constructing an updated classification model M(x) using the extended training data documents Dc; determining whether a stopping criteria has been met; based on the determination that the stopping criteria has not been met, repeating the performing step, the third constructing step, and the determining step; and, based on the determination that the stopping criteria has been met, returning the updated classification model M(x). 11. The method of claim 10, wherein the classification model, Mc, comprises one of the following: a support vector machine model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model, or ensemble model 12. The method of claim 10, wherein the Diversity Sampler process comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), and a similarity threshold t; sorting the unlabeled set of available documents D using a sorting function to obtain sorted indices I of the unlabeled set of available documents D; inserting the sorted document having a nearest sorted indice I[1] of the sorted unlabeled set of available documents D into the new batch of unlabeled instances Bc; obtaining sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; removing the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and repeating the inserting step, the second obtaining step, and the removing step until insert k documents into the new batch of unlabeled instances Bc. 13. The method of claim 12, wherein the sorting function is one of the following: a distance from a hyperplane based on smallest to greatest when the classification model Mc(x) is a support vector machine, SVM, classification model; a distance from a hyperplane based on greatest to smallest when the classification model Mc(x) is a support vector machine, SVM, classification model; or a random sorting function; an entropy function; a least confidence function; or a committee disagreement function. 14. The method of claim 10, wherein the Biased Probabilistic Sampler process comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), a similarity threshold t, and a probability distribution function P(x); constructing a probability vector w based on an probability distribution function P(x) for each document from the unlabeled set of available documents D; choosing a document I[1] from the unlabeled set of available documents D using the weight w of the probability vector; inserting the chosen document I[1] into the new batch of unlabeled instances Bc; obtaining sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; removing the documents with the obtained sorted indices I from the unlabeled set of available documents D; re-normalizing the weight vector as documents have been removed from the sorted unlabeled set of available documents D; and, repeating the choosing step, the inserting step, the second obtaining step, the removing step, and the re-normalizing step until select the new batch of unlabeled instances Bc. 15. The method of claim 10, wherein the step of determining whether the stopping criteria has been met further comprises implementing a Kappa agreement which is a measure of how much the current version of the hyperplane hc(x) and the updated hyperplane h(x) agree on respective prediction of labels of a chosen set of the documents D. 16. The method of claim 10, wherein the step of determining whether the stopping criteria has been met further comprises implementing an Accumulated History of Scores process which comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, an accumulation function A(Sc, Sc-1, . . . , S1), and a stopping threshold tstop; constructing a score vector Sc using the current classification model Mc(x) and the total set of documents DT; combining a current score vector Sc with previous score vectors (Sc-1, . . . , S1) using the accumulation function A(Sc, Sc-1, . . . , S1) to obtain a stability value (s); comparing the stability value s to the stopping threshold tstop to determine whether tstop≦S which indicates that the stopping criteria has been met; and based on the determination that the stopping criteria has not been met, storing the current score vector Sc to memory as a previous score vector Sc-1. 17. The method of claim 10, wherein the step of determining whether the stopping criteria has been met further comprises implementing a Cohen's Kappa process which comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, and a stopping threshold tstop; constructing a score vector Sc using the current classification model Mc(x) and the total set of documents DT; retrieving a previous score vector Sc-1; obtaining a set of documents D+c from the documents DT that have a positive score in Sc; obtaining a set of documents D−c from the documents DT that have a negative score in Sc obtaining a set of documents D+(c-1) from the documents DT that have a positive score in Sc-1; obtaining a set of documents D−(c-1) from the documents DT that have a negative score in Sc-1; obtaining a set of documents in common, D+, between D+c and D+(c-1); obtaining a set of documents in common, D−, between D−c and D−(c-1); obtaining a probability, P+, of a document having positive score in both score vectors Sc and Sc-1 by counting a number of documents, N+, in D+ divided by the total number of documents, N, in D; obtaining a probability, P−, of a document having negative score in both score vectors Sc and Sc-1, by counting a number of documents, N−, in D− divided by the total number of documents, N, in D; obtaining a value Ao as P++P−; obtaining a probability, P+c, by counting a number of documents, N+c, in D+c divided by the number of documents, N, in D; obtaining a probability, P−c, by counting a number of documents, N−c, in D−c divided by the number of documents, N, in D; obtaining a probability, P+(c-1), by counting a number of documents, N+(c-1), in D+(c-1) divided by the number of documents, N, in D; obtaining a probability, P−(c-1), by counting a number of documents, N−(c-1), in D−(c-1) divided by the number of documents, N, in D; obtaining a value Ae as a probability of obtaining a positive document, P+c*P+(c-1), plus a probability of obtaining a negative document, P−c*P−(c-1); obtaining a Kappa value L as Ao−Ae divided by (1−Ao); comparing the Kappa value L to the stopping threshold tstop to determine whether tstop≦Ao−Ae/1−Ae which indicates that the stopping criteria has been met; and, based on the determination that the stopping criteria has not been met, storing the score vector Sc to memory as a previous score vector Sc-1. 18. The method of claim 10, further comprising: discriminating, using the updated hyperplane h(x), remaining documents from the unlabeled set of available documents D where each of the remaining documents are classified as relevant or non-relevant. 19. An apparatus configured to implement batch-mode active learning for technology-assisted review (TAR) of documents, the apparatus comprising: a processor; and, a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the apparatus is operable to: obtain an unlabeled set of documents D; obtain a batch size k; construct a first batch of k documents D; obtain labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; construct a classification model, Mc, using the labeled first batch of k documents D; perform an iteration of active learning using the classification model Mc, wherein the perform operation comprises: (i) select a new batch of unlabeled instances Bc using a current version of the classification model, Mc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; construct an updated classification model M(x) using the extended training data documents Dc; determine whether a stopping criteria has been met, wherein the apparatus is further operable to determine whether the stopping criteria has been met by implementing an Accumulated History of Scores process which comprises the following operations: (i) obtain the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, an accumulation function A(Sc, Sc-1, . . . , S1), and a stopping threshold tstop; (ii) construct a score vector Sc using the current classification model Mc(x) and the total set of documents DT; (iii) combine a current score vector Sc with previous score vectors (Sc-1, . . . , S1) using the accumulation function A(Sc, Sc-1, . . . , S1) to obtain a stability value (s); (iv) compare the stability value s to the stopping threshold tstop to determine whether tstop≦S which indicates that the stopping criteria has been met; and (v) based on the determination that the stopping criteria has not been met, store the current score vector Sc to memory as a previous score vector Sc-1 based on the determination that the stopping criteria has not been met, repeat the perform operation, the third construct operation, and the determine operation; and, based on the determination that the stopping criteria has been met, return the updated classification model M(x). 20. The apparatus of claim 19, wherein the classification model, Mc, comprises one of the following: a support vector machine model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model, or ensemble model. 21. The apparatus of claim 19, wherein the apparatus is further operable to implement a Diversity Sampler process to select the new batch of unlabeled instances Bc as follows: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), and a similarity threshold t; sort the unlabeled set of available documents D using a sorting function to obtain sorted indices I of the unlabeled set of available documents D; insert the sorted document having a nearest sorted indice I[1] of the sorted unlabeled set of available documents D into the new batch of unlabeled instances Bc; obtain sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; remove the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and repeat the insert operation, the second obtain operation, and the remove operation until insert k documents into the new batch of unlabeled instances Bc. 22. The apparatus of claim 21, wherein the sorting function is one of the following: a distance from a hyperplane based on smallest to greatest when the classification model Mc(x) is a support vector machine, SVM, classification model; a distance from a hyperplane based on greatest to smallest when the classification model Mc(x) is a support vector machine, SVM, classification model; or a random sorting function; an entropy function; or a least confidence function; or a committee disagreement function. 23. The apparatus of claim 19, wherein the apparatus is further operable to implement a Biased Probabilistic Sampler process to select the new batch of unlabeled instances Bc as follows: obtain the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), a similarity threshold t, and a probability distribution function P(x); construct a probability vector w based on an probability distribution function P(x) for each document from the unlabeled set of available documents D; choose a document I[1] from the unlabeled set of available documents D using the weight w of the probability vector; insert the chosen document I[1] into the new batch of unlabeled instances Bc; obtain sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; remove the documents with the obtained sorted indices I from the unlabeled set of available documents D; re-normalize the weight vector as documents have been removed from the sorted unlabeled set of available documents D; and, repeat the choose operation, the insert operation, the second obtain operation, the remove operation, and the re-normalize operation until select the new batch of unlabeled instances Bc. 24. The apparatus of claim 19, wherein the apparatus is further operable to: discriminate, using the updated hyperplane h(x), remaining documents from the unlabeled set of available documents D where each of the remaining documents are classified as relevant or non-relevant. 25. A method in an apparatus for implementing batch-mode active learning for technology-assisted review (TAR) of documents, the method comprising: obtaining an unlabeled set of documents D; obtaining a batch size k; constructing a first batch of k documents D; obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; constructing a classification model, Mc, using the labeled first batch of k documents D; performing an iteration of active learning using the classification model Mc, wherein the perform operation comprises: (i) selecting a new batch of unlabeled instances Bc using a current version of the classification model, Mc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtaining labels for the new batch of unlabeled instances Bc; (iii) adding the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; constructing an updated classification model M(x) using the extended training data documents Dc; determining whether a stopping criteria has been met, wherein the apparatus is further operable to determine whether the stopping criteria has been met by implementing an Accumulated History of Scores process which comprises the following steps: (i) obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D and the extended training data documents Dc referred to as the total set of documents DT, an accumulation function A(Sc, Sc-1, . . . , S1), and a stopping threshold tstop; (ii) constructing a score vector Sc using the current classification model Mc(x) and the total set of documents DT; (iii) combining a current score vector Sc with previous score vectors (Sc-1, . . . , S1) using the accumulation function A(Sc, Sc-1, . . . , S1) to obtain a stability value (s); (iv) comparing the stability value s to the stopping threshold tstop to determine whether tstop≦S which indicates that the stopping criteria has been met; and (v) based on the determination that the stopping criteria has not been met, storing the current score vector Sc to memory as a previous score vector Sc-1 based on the determination that the stopping criteria has not been met, repeating the performing step, the third constructing step, and the determining step; and, based on the determination that the stopping criteria has been met, returning the updated classification model M(x). 26. The method of claim 25, wherein the classification model, Mc, comprises one of the following: a support vector machine model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model, or ensemble model. 27. The method of claim 25, wherein the step of selecting the new batch of unlabeled instances Bc further comprising implementing a Diversity Sampler process which comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), and a similarity threshold t; sorting the unlabeled set of available documents D using a sorting function to obtain sorted indices I of the unlabeled set of available documents D; inserting the sorted document having a nearest sorted indice I[1] of the sorted unlabeled set of available documents D into the new batch of unlabeled instances Bc; obtaining sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; removing the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and repeating the inserting step, the second obtaining step, and the removing step until insert k documents into the new batch of unlabeled instances Bc. 28. The apparatus of claim 27, wherein the sorting function is one of the following: a distance from a hyperplane based on smallest to greatest when the classification model Mc(x) is a support vector machine, SVM, classification model; a distance from a hyperplane based on greatest to smallest when the classification model Mc(x) is a support vector machine, SVM, classification model; or a random sorting function; an entropy function; a least confidence function; or a committee disagreement function. 29. The method of claim 25, wherein the step of selecting the new batch of unlabeled instances Bc further comprising implementing a Biased Probabilistic Sampler process which comprises the following steps: obtaining the current version of the classification model Mc(x), the unlabeled set of available documents D, the batch size k, a similarity function S(x1, x2), a similarity threshold t, and a probability distribution function P(x); constructing a probability vector w based on an probability distribution function P(x) for each document from the unlabeled set of available documents D; choosing a document I[1] from the unlabeled set of available documents D using the weight w of the probability vector; inserting the chosen document I[1] into the new batch of unlabeled instances Bc; obtaining sorted indices I of the sorted unlabeled set of available documents D that have a similarity score S(XI[1], X)≧t with respect to the inserted document I[1]; removing the documents with the obtained sorted indices I from the unlabeled set of available documents D; re-normalizing the weight vector as documents have been removed from the sorted unlabeled set of available documents D; and, repeating the choosing step, the inserting step, the second obtaining step, the removing step, and the re-normalizing step until select the new batch of unlabeled instances Bc. 30. The method of claim 25, further comprising: discriminating, using the updated hyperplane h(x), remaining documents from the unlabeled set of available documents D where each of the remaining documents are classified as relevant or non-relevant.