Crowdsourcing systems with machine learning are described, for example, to aggregate answers to a crowdsourced task in a manner achieving good accuracy even where observed data about past behavior of crowd members is sparse. In various examples a machine learning system jointly learns variables describing characteristics of both individual crowd workers and communities of the workers. In various examples, the machine learning system learns aggregated labels. In examples learnt variables describing characteristics of an individual crowd worker are related, by addition of noise, to learnt variables describing characteristics of a community of which the individual is a member. In examples the crowdsourcing system uses the learnt variables describing characteristics of individual workers and of communities of workers for any one or more of: active learning, targeted training of workers, targeted issuance of tasks, calculating and issuing rewards.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A computer-implemented method of labeling items comprising: issuing, over a communications network, a labeling task to a plurality of workers participating in a crowdsourcing system; receiving, over the communications network, labels assigned to the issued task by two or more of the workers; sending the received labels and task data to inference components arranged to jointly infer both worker characteristics and characteristics of worker communities of which the workers are members, by assuming relationships between workers; and receiving inference results from the inference components, the inference results comprising an aggregated label for the task, the aggregated label taking into account the inferred worker characteristics and the inferred worker community characteristics. 2. A method as claimed in claim 1 wherein receiving inference results further comprises receiving inferred worker community characteristics and using the inferred worker community characteristics to select workers from the plurality of workers for a future task. 3. A method as claimed in claim 1 wherein receiving inference results comprises receiving uncertainty information associated with the inference results and using the uncertainty information to identify future task and worker combinations to issue. 4. A method as claimed in claim 1 wherein receiving inference results further comprises receiving inferred worker community characteristics and using the inferred worker community characteristics to select workers from the plurality of workers for training. 5. A method as claimed in claim 1 wherein receiving inference results comprises receiving a probability distribution describing belief about the value of an aggregated label. 6. A method as claimed in claim 1 comprising sending the received labels and task data to inference components which relate the community characteristics to the worker characteristics through the addition of noise. 7. A method as claimed in claim 1 comprising carrying out inference at the inference components in order to update probability distributions representing belief about latent variables representing the community characteristics, the worker characteristics and the aggregated label. 8. A method as claimed in claim 7 comprising carrying out the inference using inference algorithms implementing either variational message passing or Expectation Propagation. 9. A method as claimed in claim 7 comprising carrying out the inference using inference algorithms compiled from a probabilistic graphical model. 10. A method as claimed in claim 7 wherein the latent variables representing the community characteristics and the worker characteristics comprise confusion matrices. 11. A method as claimed in claim 7 comprising using prior knowledge to inform the inference at the inference components, the prior knowledge comprising any one or more of: a number of the worker communities, composition of the worker communities. 12. A method as claimed in claim 1 comprising training the inference components using any one of more of: online training, offline training, online batch training. 13. A method as claimed in claim 1 comprising sending the received labels and task data to inference components comprising inference algorithms compiled from a probabilistic graphical model, the probabilistic graphical model being divided into a plurality of sub-models, such that local inference may be carried out on the basis of each sub-model and the local inference results merged. 14. A method as claimed in claim 13 comprising carrying out the inference at the inference components using parallel processing of the sub-models. 15. A computer-implemented method of labeling items comprising: issuing, over a communications network, a labeling task to a plurality of workers participating in a crowdsourcing system; receiving, over the communications network, labels assigned to the issued task by two or more of the workers; carrying out inference using the received labels and task data to jointly infer both worker characteristics and characteristics of worker communities of which the workers are members, by assuming relationships between workers; and also inferring an aggregated label for the task, the aggregated label taking into account the inferred worker characteristics and the inferred worker community characteristics. 16. A crowdsourcing system comprising: a communication interface arranged to issue, over a communications network, a labeling task to a plurality of workers participating in the crowdsourcing system; the communications interface arranged to receive, over the communications network, labels assigned to the issued task by two or more of the workers; a processor arranged to send the received labels and task data to inference components arranged to jointly infer both worker characteristics and characteristics of worker communities of which the workers are members, by assuming relationships between workers; and the processor arranged to receive inference results from the inference components, the inference results comprising an aggregated label for the task, the aggregated label taking into account the inferred worker characteristics and the inferred worker community characteristics. 17. A crowdsourcing system as claimed in claim 16 wherein the processor is arranged to select workers from the plurality of workers for a future task on the basis of the worker community characteristics. 18. A crowdsourcing system as claimed in claim 16 wherein the processor is arranged to select workers and generate tasks on the basis of certainty of the worker community characteristics. 19. A crowdsourcing system as claimed in claim 16 wherein the processor is arranged to select worker communities for training on the basis of the worker community characteristics. 20. A crowdsourcing system as claimed in claim 16 comprising the inference components and wherein the inference components are at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.