Crowdsourcing is the phenomenon of outsouring a problem to the public in order to get diverse solutions to that problem . Crowdsouring Software Development is a contest based activity in which organizations use crowdsourcing platforms to float a problem related to any phase of Software Development Life Cycle . Volunteers participate and compete for this activity and best solution provider gets the reward . In building on this theme the author explores the role that crowdsourcing can plan in improving metadata quality in the current article . The author reviews an “ experiment ” in using crowdsourcing platforms to improve metadata quality . The platforms/metadata quality issues tested were used to validate technical metadata on proxy configuration . However , a major challenge is to achieve the reliability this information within datasets so heterogeneous or complex . For instance , reports of Volunteered Geographic Information ( VGI ) through a crowdsourcing-based platform can be confirmed by means of images available on site . Thus , we carried out an experiment to identify water level of river through clustering from static images using an evolutionary method of hierarchical data clustering , called DAta-MIning COde REpositories 1 ( Sanches , Cardoso , and Delbem , 2011 ) . Existing smart city systems are designed mainly for urban management from a governmental perspective , but they fail to cover the wide spectrum of citizens ' daily lives . We propose a crowdsourcing-based platform through which multiple players can collaborate to produce more abundant , personalized , and proactive services . However , due to lack of a central mechanism for team formation , continuity in the developer 's work on consecutive tasks and risk of noise in submissions of a contest , requesters of a domain have quality concerns to adopt a crowdsourcing-based software development platform . In order to address this concern , we proposed a measure Quality of Contest ( QoC ) to analyze and predict the quality of a crowdsourcing-based platform through historical information on its completed tasks . We evaluate the capacity of QoC as assessor to predict the quality . Crowdsourcing product development is a flexible and efficient solution for product design and manufacture . At present , crowdsourcing community platform for micro task is mainly based on Web2.0 , for crowdsourcing product development , it is an urgent issue to figure out how to utilize a massive quantity of service resource combination in a cross-platform manner in the cloud crowdsourcing environment . By analyzing the influence of big data on crowdsourcing product development based on cloud , this paper proposes a modeling method for optimization of cloud-service resource combination for crowdsourcing product development , using the cooperative bacterial foraging optimization ( CBFO ) algorithm . This is a complex problem requiring sophisticated solutions balanced with the need to present sometimes unsophisticated users with comprehensible and useable software . COBWEB has addressed this challenge by using the UNESCO World Network of Biosphere Reserves as a testbed for researching and developing a generic crowdsourcing infrastructure platform for environmental monitoring . The solution arrived at provides tools for the creation of mobile Applications which generate data compliant with open interoperability standards and facilitate integration with Spatial Data Infrastructures . The research will collect survey data regarding enterprise readiness , and will examine the current state of enterprise readiness to adopt new development techniques . Study three will utilize a design science perspective to examine the ability of crowdsourcing marketplace platforms to meet the needs of IT service buyers and IT service workers as identified in Study 's one and two . However , it is widely known that deep architectures are hungry of data . Therefore , we addressed the problem of how to collect more labelled plant image datasets , using three approaches : ( i ) we developed an annotation tool to help experts to annotate images ; ( ii ) we uploaded images in a crowdsourcing online platform , allowing citizen scientists to annotate them ; ( iii ) we used a generative deep neural network to synthesise the images of plants with the leaf count . Lastly , we will show how a deep leaf counting network can be trained with data from different sources and modalities , showing promising results and reducing the performance gap between algorithm and human annotators . This article presents the adaptation to Guadeloupean Creole of a project of crowdsourcing part-of-speech ( POS ) tags initially designed for a French regional language , Alsatian . We do not detail here the specifically developed crowdsourcing platform and methodology , but rather focus on the construction of the required elements for a language to be a candidate for this task : i ) an open-source raw corpus , ii ) a tokenizer , iii ) adapted annotation guidelines , iv ) a minimal reference , and , preferentially , v ) one or two baseline tagger(s ) . After describing the preliminary work we have carried out for Guadeloupean Creole to comply with these prerequisites , we present the first results on crowdsourcing POS tags through the platform specifically developed for this task : Krik . ” Requirement pattern is one of the essences for the cognitive service to understand customer ’s intention . Since crowdsourcing service platform holds abundant user demands in the form of text information , the method proposed in this paper aims at eliciting valuable patterns from this “ treasure . ” section : Related Work We briefly review two lines of research relevant to the current effort : work on NLP to facilitate EBM , and research in crowdsourcing for NLP . subsection : NLP for EBM A solution is to develop a data generation pipeline . Once the scope of an assistant has been defined , a mix of crowdsourcing and semi - supervised machine learning is used to generate thousands of high - quality training examples , mimicking user data collection without compromising on privacy . The aim of this section is to describe the data generation pipeline , and to demonstrate its impact on the performance of the NLU . Widespread use of advanced mobile devices has led to the emergence of a new class of crowdsourcing called spatial crowdsourcing . Spatial crowdsourcing advances the potential of a crowd to perform tasks related to real-world scenarios involving physical locations , which were not feasible with conventional crowdsourcing methods . The main feature of spatial crowdsourcing is the presence of spatial tasks that require workers to be physically present at a particular location for task fulfillment . Research related to this new paradigm has gained momentum in recent years , necessitating a comprehensive survey to offer a bird’s-eye view of the current state of spatial crowdsourcing literature . In this article , we discuss the spatial crowdsourcing infrastructure and identify the fundamental differences between spatial and conventional crowdsourcing . Furthermore , we provide a comprehensive view of the existing literature by introducing a taxonomy , elucidate the issues/challenges faced by different components of spatial crowdsourcing , and suggest potential research directions for the future . In most cases , existing approaches usually take the matching algorithm as a fundamental step to solve this problem which aims to maximize the number of completed tasks . However , in the present of many spatial crowdsourcing platforms , how to assign the tasks at high efficiency and make a relatively fair schedule for multiple workers is a new challenge . In this paper , we study the problem of load balancing based task scheduling for multiple workers . With the rapid development of Mobile Internet and Online To Offline ( O2O ) marketing model , various spatial crowdsourcing platforms , such as Gigwalk and Gmission , are getting popular . Most existing studies assume that spatial crowdsourced tasks are simple and trivial . Spatial Crowdsourcing Platforms ( SCP ) are systems that act as brokers that help people requesting location dependent tasks to find suitable workforce to accomplish it . This kind of system has been source of much interest for both academy and industry . This list has been identified and validated by an analysis of scientific related works and several commercial SCPs . Based on this list , we also propose a generic architecture for SCPs that is meant to help the understanding of SCPs and the design of better spatial crowdsourcing solutions . Spatial crowdsourcing services , such as Uber and Grabhub , become popular recently . Task assignment plays an important role in offering high-quality services . Under the premise that an intrinsic motivation of contestants on crowdsourcing platforms is to solve challenging problems , a theoretical model of the relationships between the two measures and the contest outcome , mediated by the quality of contestants , is developed . Data from a popular software crowdsourcing platform , TopCoder , is used to validate hypotheses on the direct , interaction , and nonlinear impacts of the quality level and quality ambiguity of a prior phase contest ’s winning solution on the outcome of the focal phase contest . Results show an inverted U-shaped relationship between the quality ambiguity of a prior phase contest ’s winning solution and the quality of the contestants in the focal phase contest , highlighting a sweet spot for quality ambiguity . how to leverage Git 's powerful yet underused advanced features . The last Web extra at http://youtu.be/SK6TBI1bNLI is a video of Thomas Baden , Chief Information Officer , State of Minnesota , Department of Human Services , describing the experience of working on the TopCoder Platform and with the members of the TopCoder Community . First , we derive a set of key concerns from the crowdsourcing literature as an initial analytical framework for an exploratory case study in a Fortune 500 company . We complement the case study findings with an analysis of 13,602 crowdsourcing competitions over a ten-year period on the very popular Topcoder crowdsourcing platform . Drawing from our empirical findings and the crowdsourcing literature , we propose a theoretical model of crowd interest and actual participation in crowdsourcing competitions . Research to date on crowdsourcing software development has tended to focus on the ' crowd ' or the technical platform , with little research from the perspective of the customer who is seeking to leverage the crowdsourcing development model . The findings from an in-depth case study of crowd-sourcing software development in a Fortune 500 company are augmented with an analysis of over 13,000 crowdsourcing competitions over a ten-year period on the Topcoder crowdsourcing platform , one of the most popular platforms for software development , are drawn on to evaluate the effectiveness of crowdsourcing in a software development context We justify a definition of performance using the logarithm of a player 's rank . We apply the definition to rating TopCoder SRM . We improve the accuracy , guided by experimental results . Presenting an algorithm to construct worker competition network and extract its network metrics features . The proposed framework was evaluated on 4,872 software crowdsourcing tasks , extracted from TopCoder platform , using five machine learners , compared with in-house TopCoder predictor . Results : 1 ) Workers reliability , links in the description , number of registered workers , number of required technologies , and task-workers network modularity are the most influencing factors for predicting crowdsourcing failure ; 2 ) The top-three learners for task failure are Naïve Bayes , Random Forest , and StackingC , with precision above 98.8 % , recall above 81.2 % , and F-measure above 91.2 % ; and 3 ) The proposed best learners significantly outperform the two baseline models in our evaluation . Some examples include problems that require more than one data structure and/or algorithm , problems that require modifications to standard data structures and/or algorithms , and problems that require attention to efficiency both in time and space . The Topcoder competitive coding platform can be an excellent resource to provide these examples and can be used in several ways in the classroom . In this paper , we detail our experience with Topcoder in several classes and from several perspectives . The prize of each task may be determined in terms of the expected duration of the task , its complexity , its quality , the required effort to complete , and the online reputation of the client . TopCoder awarding mechanism , performance-based , game theory-based , deadline driven or revenue sharing models are several approaches for determining task prizes in crowdsourced software development ( CSD ) . In this paper , Putnam model , a well-known software effort estimation model , is proposed to be applicable as a prize determination approach in crowdsourcing . To develop the web-based tool , the HH&P utilized the NASA Tournament Lab ( NTL ) to develop the software with TopCoder under an existing contract . In this way , the HH&P is using one new tool ( the NTL and TopCoder ) to develop the next generation tool , the SMG . The beta-phase of the SMG is planed for release in the spring of 2014 and results of the beta-phase testing will be available for the IAC meeting in September . The paper then defines a game theory model to analyze the primary factors in these minmax competition rules that affect the nature of participation as well as the software quality . Finally , using the proposed evaluation framework , this paper illustrates two crowdsourcing processes , Harvard-TopCoder and AppStori . The framework demonstrates the sharp contrasts between both crowdsourcing processes as participants will have drastic behaviors in engaging these two projects . Such sharing , however , is at tension with the competitive nature of crowd contests . To understand this tension , this study investigates community-level sharing of code on Kaggle , a contest platform for predictive modeling . Analyzing data on 25 contests in 2015 and 2016 , we find that 10 % of users shared code during contests , that participants doing medium well in the contest were the most likely to share code , and that sharing code improved individual , but not collective performance . The topic of the 2014 ALTA shared task was to identify location information in tweets . As in past competitions , we used Kaggle in Class as the framework for submission , evaluation and communication with the participants . In this paper we describe the details of the shared task , evaluation method , and results of the participating systems . This paper describes Task 2 of the DCASE 2018 Challenge , titled " General-purpose audio tagging of Freesound content with AudioSet labels " . This task was hosted on the Kaggle platform as " Freesound General-Purpose Audio Tagging Challenge " . The goal of the task is to build an audio tagging system that can recognize the category of an audio clip from a subset of 41 diverse categories drawn from the AudioSet Ontology . Coupled with transfer learning and hyper-parameter tuning , we adopt AlexNet , VggNet , GoogleNet , ResNet , and analyze how well these models do with the DR image classification . We employ publicly available Kaggle platform for training these models . The best classification accuracy is 95.68 % and the results have demonstrated the better accuracy of CNNs and transfer learning on DR image classification . A solution to the Kaggle competition : Human Protein Atlas Image Classification ( HPA ) . Using microscopic images of cells provided by the Human Protein Atlas , convolutional neural networks , CNNs , were used to analyze and predict the location of protein patterns . Challenges included working with an unbalanced dataset , finding a correct learning rate , and choosing a correct architecture to solve the problem . Background Kaggle is an online platform for predictive modelling and analytics competitions , that hosts the HPA competition . In this competition , competitors try to build models to classify mixed patterns of proteins found in microscopic images . This document describes the models and software used to build our solution . Reproducing the results for our solution can be done by running the scripts in-cluded in the Kaggle package et al. ( 2011 ) . We show that our automated approach performs better than the best performing Kaggle entry and generalizes as a method to the Mohler dataset . Hence , University of Texas MD Anderson Cancer Center organized two public radiomics challenges in head and neck radiation oncology domain . This was done in conjunction with MICCAI 2016 satellite symposium using Kaggle-in-Class , a machine-learning and predictive analytics platform . We drew on clinical data matched to radiomics data derived from diagnostic contrast-enhanced computed tomography ( CECT ) images in a dataset of 315 patients with oropharyngeal cancer . LL-CNN was compared with 2D-UNet , 3D-UNet , a single-task CNN ( ST-CNN ) , and a pure multitask CNN ( MT-CNN ) . Training , validation , and testing followed Kaggle competition rules , where 160 patients were used for training , 20 were used for internal validation , and 20 in a separate test set were used to report final prediction accuracies . RESULTS On average contours generated with LL-CNN had higher Dice coefficients and lower RMSE than 2D-UNet , 3D-Unet , ST- CNN , and MT-CNN . For the application , we worked with 3168 recorded voices , for which the results of an acoustic analysis are presented , registering variables such as frequency , spectrum , modulation , among others , seeking to obtain a pattern of identification and classification according to gender through a voice identifier . The data record used is in open access and can be downloaded from the Kaggle web platform via r. For the development of the algorithm ’s model , the statistical program R was used . Additionally , applications were made with Python by the development of classification algorithms . Using five-fold cross-validation , we train a 3D attention-based network under two scenarios : ( 1 ) single-task learning with only classification , and ( 2 ) multi-task learning with both classification and regression . The single-task learning leads to a higher AUC compared with the Kaggle challenge winner pre-trained model ( 0.878 v. 0.856 ) , and multi-task learning significantly improves the single-task one ( AUC 0.895 , p<0.01 , McNemar test ) . In summary , the image-based predicted CFPT can be used in follow-up year lung cancer prediction and data assessment . Study findings indicate that online crowdsourcing , a process that allows access to many participants to complete specific tasks , is a potentially viable resource for evaluation designs where access to comparison groups , large budgets , and/or time are limited . The article highlights the strengths and limitations of the online crowdsourcing approach and describes ways that it could potentially be used in evaluation practice . Mobile and social technologies are providing new opportunities to document , characterize , and gather impressions of urban environments . In this article , we present a study that examines urban perceptions of three cities in central Mexico ; the study integrates a mobile crowdsourcing framework to collect geo-localized images of urban environments by a local youth community , an online crowdsourcing platform to gather impressions of urban environments along 12 physical and psychological dimensions , and a deep learning framework to automatically infer human impressions of outdoor urban scenes . Our study resulted in a collection of 7,000 geo-localized images containing outdoor scenes and views of each city ’s built environment , including touristic , historical , and residential neighborhoods , and 144,000 individual judgments from Amazon Mechanical Turk . ABSTRACT The objective of this study is to explore the factors that are keys for an idea to be implemented on an online crowdsourcing platform . A data set of 320 implemented ideas from My Starbucks Idea – an online crowdsourcing platform – has been analyzed . We find that only one out of 500 users ’ submitted ideas are selected for implementation . The objective of this study is to explore the factors that are keys for an idea to be implemented on an online crowdsourcing platform . A data set of 320 implemented ideas from My Starbucks Idea – an online crowdsourcing platform – has been analyzed . We find that only one out of 500 users ’ submitted ideas are selected for implementation . A growing interest in using crowdsourcing to collect and annotate datasets has been observed in recent years . In this paper , we have implemented a toolbox to easily develop online crowdsourcing tools to build a corpus of virtual agent ’s non-verbal behaviors directly rated by users . We present two developed online crowdsourcing tools that have been used to construct a repertoire of virtual smiles and to define virtual agents ’ non-verbal behaviors associated to social attitudes . In this paper we investigate to what extent online crowdsourcing can be used for the specific case of safety perceptions about people . We built an image – based online crowdsourcing platform , collected safety perception ratings and background information from more than 700 people and used them to quantitatively evaluate established theories based on qualitative research . On one hand , we show in this paper that online , image-based crowdsourcing can be used to gather perceptions about people too , not just architecture , confirming established theories based on qualitative work . The successful execution of knowledge crowdsourcing ( KC ) tasks requires contributors to possess knowledge or mastery in a specific domain . The need for expert contributors limits the capacity of online crowdsourcing marketplaces to cope with KC tasks . While online social platforms emerge as a viable alternative source of expert contributors , how to successfully invite them remains an open research question . While online Crowdsourcing marketplaces provide a powerful avenue for facilitating new forms of information- driven micro-labor , their practical value is significantly reduced by worker “ spam ” and employer fraud . We hypothesize anonymity of parties is a major source of these problems , and we thus propose a human-centric solution : encourage employers and workers to voluntarily de- anonymize in order to reap a potential benefit of more productive and profitable labor interactions . Mobile crowdsourcing markets ( e.g. , Gigwalk and TaskRabbit ) offer crowdworkers tasks situated in the physical world ( e.g. , checking street signs , running household errands ) . The geographic nature of these tasks distinguishes these markets from online crowdsourcing markets and raises new , fundamental questions . We carried out a controlled study in the Chicago metropolitan area aimed at addressing two key questions : ( 1 ) What geographic factors influence whether a crowdworker will be willing to do a task ? To do so we carried out two studies where participants performed 3 communication tasks , solo in study one and dyadic in study 2 , and were recorded on video both with and without robot mediation . Judges recruited using online crowdsourcing services then made personality judgements of those in the video clips . We observed that judges were able to make internally consistent trait judgements in both communication conditions . Several software crowdsourcing models performed on typical platforms like Topcode , uTest are established and their organization and coordination are discussed . We have developed a service-based multi-agent system platform called AutoService that provides some fundamental capabilities like autonomy , monitoring , flexible interaction and organization , and can serve as an infrastructure to support software crowdsourcing models and tackle its challenges . A software crowdsourcing prototype is developed and some scenarios are exemplified to illustrate our approach . With the increasing role of software crowdsourcing , a new working architecture for software development , in global software development ( GSD ) settings , has emerged challenges regarding sustainable software development and software sustainability , which are still to be elucidated . Software standards and models exist to assist co-located and global software development ; however , seismic changes in the working architecture of software development as software crowdsourcing are highlighting a risk of software failure . Recently , statistics showed that only 30 % of projects are successful , whereas 50 % encounter challenges and the remaining 20 % are absolute failures that implicate a challenge toward software success and its sustainability . Thus , a lack of any guideline toward sustainable software crowdsourcing is highlighted as one of the limitations in the software standards . This paper contributes to the GSD industry in two folds , first by laying down the importance and theoretical association on and around sustainable crowdsourcing , and second by highlighting that the IEEE standards , inadequacy toward the suppressing phenomena of sustainability in software crowdsourcing , if addressed adequately , can minimize the ripple effect toward software failure . However , existing rating systems are facing challenges such as providing limited information regarding worker 's abilities as well as potential threats from workers ' gaming or cheating the systems . To develop better understanding of worker performance in software crowdsourcing , this paper reports an empirical study at TopCoder , one of the primary software crowdsourcing platforms . We aim at investigating the following questions : How diverse are crowd workers in terms of skill and experience ? The key idea of CrowDevBot is to : ( 1 ) combine a rulebased method and an SVM-NaiveBayes-C4.5 integrated learning method to discover users ’ intention ; ( 2 ) employ an integrated CRF ( conditional random field ) method with novel features to improve the performance of slot filling ; and ( 3 ) leverage a software service knowledge base to unify entity names and predefine the key slots of user query . We implement CrowDevBot and integrate it into JointForce , an IT software crowdsourcing platform in China . To the best of our knowledge , this is the first time that a task-oriented conversational bot is practically used in software crowdsourcing platform(s ) . more relevant context , i.e. a set of similar tasks identified based on Topic modeling and we evaluate 7 predictive models . The proposed approach is evaluated on 450 software tasks extracted from TopCoder , the largest software crowdsourcing platform . The results show that : 1 ) the proposed models can be used at early crowdsourcing planning phase , when information on traditional metrics are not available ; 2 ) the best model achieves 65 % in accuracy measure ; 3 ) the local model exhibits 27 % increases superior to the global model . The Gene Wiki has created over 10,000 Wikipedia articles , and aims to provide one for every notable human gene [ 10 ] . The crowdsourcing platform InnoCentive has reportedly facilitated solutions to roughly half of the thousands of technical problems posed on the site , including many in life sciences such as the $ 1 million ALS Biomarker Prize [ 11 ] . Other examples include prizes ( X-Prize [ 12 ] ) , scientific games ( FoldIt [ 13 ] ) , and licensing schemes inspired by open-source software ( BIOS [ 14 ] ) . Collaborative R&D approaches vary in openness [ 15 ] . In some approaches , the R&D process and outputs are open to all — for example , open-science projects like the Gene Wiki described above . In other approaches which demonstrate what might be called controlled collaboration , there are strong controls on who contributes and benefits — for example , computational platforms like Collaborative Drug Discovery or InnoCentive that support both commercial and nonprofit research [ 9 ] , [ 11 ] . Collaborative approaches can unleash innovation from unforeseen sources , as with crowdsourcing health technologies [11]–[13 ] , [ 16 ] . In fact , spatial crowdsourcing has stimulated a series of recent industrial successes including Citizen Sensing ( Waze ) , P2P ride-sharing ( Uber ) and Real-time Online-To-Offline ( O2O ) services ( Instacart and Postmates ) . In this tutorial , we review the paradigm shift from web-based crowdsourcing to spatial crowdsourcing . We dive deep into the challenges and techniques brought by the unique spatio-temporal characteristics of spatial crowdsourcing . Although some of them utilize this approach , they use one of the web-based crowdsourcing frameworks with paid crowdworkers . In this paper we decided to present our own developed web-based crowdsourcing framework and use it as unpaid approach for evaluating two well-known compression standards - H.264 and H.265 and for comparing the results getting from testing via this crowdsourcing framework with results which we got from laboratory testing . These results showed that the subjective quality testing via unpaid crowdsourcing approach can be used as a replacement to the subjective laboratory testing , especially in cases where not very sophisticated evaluation is required . Agora aims to be an " operating system " for social media applications gathering data . Furthermore , the Web-based crowdsourcing platform metropolitalia has been established for hosting two social media applications based upon Agora : Mercato Linguistico and Poker Parole . These applications have been conceived as part of this thesis for gathering complementary data and meta-data on Italian language varieties . Our proposed solution provides a collaborative platform to acquire assessments from a larger group of experts , towards forming a collective insight and minimizing overlooking of defects . Our first-of-its-kind web-based crowdsourcing platform can be leveraged for 3D visualization of microscopy data without imposing any computational load on the users , as well as collaborative analysis by collecting information from each user . Further , the collected information is compiled in a data bank , which serves as a valuable source for developing quantified measures and for training automated defect classification algorithms . The requirements of enterprise crowdsourcing platforms are compiled based on the literature and a qualitative study , providing a conceptual design of task assignment strategies . The demands of workers and their perception of task similarity on public crowdsourcing platforms are identified , leading to the design and implementation of additional methods to determine the similarity of micro-tasks . The textual descriptions of micro-tasks , projects , and job postings are analyzed in order to provide innovative methods for task recommendation in these domains . The 5C categorization is based on the action s enabled by the social media tools : communicating , collaborating , connecting , completing , and combining . In order to demonstrate how the 5C categorization can be used as a framework to assess social media tools we conducted an empirical study of social media based crowdsourcing platforms in business-to-business setting . The 5C categorization can be utilized in planning a social media strategy as it forwards the understanding of which tools are suitable for certain purpose and provides a scheme to evaluate and compare different social media tools and applications . Crowdsourcing is a useful approach for organizing a database based on collaborative works of people . We recruited trial users to test our system via a web-based crowdsourcing service provider in Japan . However , we found that improvement of the system is important for maintaining the motivation of users in order to continue the collection of sounds . Such combination requires an understanding of where the algorithms fail , which we gather using a novel coarse scribble video annotation task . We evaluate our ensemble methods , discuss our design choices for them , and make our web-based crowdsourcing tools and results publicly available . In this paper we specifically focus on the impact of online citizen science ( OCS ) participation in the science education of primary school age children in New Zealand . Methods : We use four exploratory cases within a broader research project to examine the nature and impact of embedding OCS projects that use web based online crowdsourcing and collaboration tools within classroom environments of primary school science learners . Results & Discussion : Our findings provide insights into primary school teachers ' perception of OCS . This work examines to which extend , " trapping question " or " outliers detection " assure reliable results to the detriment of , overloading task content with stimuli that are not of interest for the researcher , or by discarding data points that might be the true opinion of a worker . To this end , a speech quality assessment study have been conducted in a web crowdsourcing platform , following the ITU-T Rec . P.800 . To reduce the time cost , we adopt split techniques and design Simple Split(SS ) and Dynamic Split(DS ) algorithms . We verify the effectiveness of our solutions by conducting crowdsourcing simulation experiments and experiments on a crowdsourcing platform namely gMission . The third paper , titled Toward an Understanding of Participants ’ sustained participation in Crowdsourcing Contests by Xuan Wang , Hanieh Javadi Khasraghi , and Helmut Schneider , addresses a little known issue regarding the factors that influence individuals ’ continued participation in crowdsourcing contests . The authors conduct an empirical study using data from an online crowdsourcing contest platform , Kaggle , which delivers data science and machine learning solutions and models to its clients . The analysis finds that both collaborative and competitive components such as tenure , performance , price amount , number of competitors , and Resource Allocation and Alignment Effects . Abstract Advances in information technology bring changes to the nature of work by facilitating companies to go beyond the wisdom of their workforce and tap into the “ wisdom of the crowd ” via online crowdsourcing contests . In these contests , active and motivated individuals collaborate in the form of self-organized teams that compete for rewards . Recent advances in information technology bring significant changes to the nature of work by allowing companies to tap “ wisdom of the crowd ” beyond their own workforce . In particular , organizations increasingly seek solutions to their business problems using online crowdsourcing contests that are open to the public . In these contests , self&organized vi rtual teams compete for monetary reward . However , there are some barriers preventing them from adopting crowdsourcing into their product design and development ( PDD ) practice . In this paper , we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools , crowdsourcing frameworks , and techniques in terms of open call generation , rewarding , crowd qualification for working , organization structure of crowds , solution evaluation , workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process . We also explore the necessary techniques and tools to support the crowdsourcing PDD process .
Crowdsourcing is the phenomenon of outsouring a problem to the public in order to get diverse solutions to that problem . Crowdsouring Software Development is a contest based activity in which organizations use crowdsourcing platforms to float a problem related to any phase of Software Development Life Cycle . Volunteers participate and compete for this activity and best solution provider gets the reward .
In building on this theme the author explores the role that crowdsourcing can plan in improving metadata quality in the current article . The author reviews an “ experiment ” in using crowdsourcing platforms to improve metadata quality . The platforms/metadata quality issues tested were used to validate technical metadata on proxy configuration .
However , a major challenge is to achieve the reliability this information within datasets so heterogeneous or complex . For instance , reports of Volunteered Geographic Information ( VGI ) through a crowdsourcing-based platform can be confirmed by means of images available on site . Thus , we carried out an experiment to identify water level of river through clustering from static images using an evolutionary method of hierarchical data clustering , called DAta-MIning COde REpositories 1 ( Sanches , Cardoso , and Delbem , 2011 ) .
Existing smart city systems are designed mainly for urban management from a governmental perspective , but they fail to cover the wide spectrum of citizens ' daily lives . We propose a crowdsourcing-based platform through which multiple players can collaborate to produce more abundant , personalized , and proactive services .
However , due to lack of a central mechanism for team formation , continuity in the developer 's work on consecutive tasks and risk of noise in submissions of a contest , requesters of a domain have quality concerns to adopt a crowdsourcing-based software development platform . In order to address this concern , we proposed a measure Quality of Contest ( QoC ) to analyze and predict the quality of a crowdsourcing-based platform through historical information on its completed tasks . We evaluate the capacity of QoC as assessor to predict the quality .
Crowdsourcing product development is a flexible and efficient solution for product design and manufacture . At present , crowdsourcing community platform for micro task is mainly based on Web2.0 , for crowdsourcing product development , it is an urgent issue to figure out how to utilize a massive quantity of service resource combination in a cross-platform manner in the cloud crowdsourcing environment . By analyzing the influence of big data on crowdsourcing product development based on cloud , this paper proposes a modeling method for optimization of cloud-service resource combination for crowdsourcing product development , using the cooperative bacterial foraging optimization ( CBFO ) algorithm .
This is a complex problem requiring sophisticated solutions balanced with the need to present sometimes unsophisticated users with comprehensible and useable software . COBWEB has addressed this challenge by using the UNESCO World Network of Biosphere Reserves as a testbed for researching and developing a generic crowdsourcing infrastructure platform for environmental monitoring . The solution arrived at provides tools for the creation of mobile Applications which generate data compliant with open interoperability standards and facilitate integration with Spatial Data Infrastructures .
The research will collect survey data regarding enterprise readiness , and will examine the current state of enterprise readiness to adopt new development techniques . Study three will utilize a design science perspective to examine the ability of crowdsourcing marketplace platforms to meet the needs of IT service buyers and IT service workers as identified in Study 's one and two .
However , it is widely known that deep architectures are hungry of data . Therefore , we addressed the problem of how to collect more labelled plant image datasets , using three approaches : ( i ) we developed an annotation tool to help experts to annotate images ; ( ii ) we uploaded images in a crowdsourcing online platform , allowing citizen scientists to annotate them ; ( iii ) we used a generative deep neural network to synthesise the images of plants with the leaf count . Lastly , we will show how a deep leaf counting network can be trained with data from different sources and modalities , showing promising results and reducing the performance gap between algorithm and human annotators .
This article presents the adaptation to Guadeloupean Creole of a project of crowdsourcing part-of-speech ( POS ) tags initially designed for a French regional language , Alsatian . We do not detail here the specifically developed crowdsourcing platform and methodology , but rather focus on the construction of the required elements for a language to be a candidate for this task : i ) an open-source raw corpus , ii ) a tokenizer , iii ) adapted annotation guidelines , iv ) a minimal reference , and , preferentially , v ) one or two baseline tagger(s ) . After describing the preliminary work we have carried out for Guadeloupean Creole to comply with these prerequisites , we present the first results on crowdsourcing POS tags through the platform specifically developed for this task : Krik .
” Requirement pattern is one of the essences for the cognitive service to understand customer ’s intention . Since crowdsourcing service platform holds abundant user demands in the form of text information , the method proposed in this paper aims at eliciting valuable patterns from this “ treasure . ”
section : Related Work We briefly review two lines of research relevant to the current effort : work on NLP to facilitate EBM , and research in crowdsourcing for NLP . subsection : NLP for EBM
A solution is to develop a data generation pipeline . Once the scope of an assistant has been defined , a mix of crowdsourcing and semi - supervised machine learning is used to generate thousands of high - quality training examples , mimicking user data collection without compromising on privacy . The aim of this section is to describe the data generation pipeline , and to demonstrate its impact on the performance of the NLU .
Widespread use of advanced mobile devices has led to the emergence of a new class of crowdsourcing called spatial crowdsourcing . Spatial crowdsourcing advances the potential of a crowd to perform tasks related to real-world scenarios involving physical locations , which were not feasible with conventional crowdsourcing methods . The main feature of spatial crowdsourcing is the presence of spatial tasks that require workers to be physically present at a particular location for task fulfillment . Research related to this new paradigm has gained momentum in recent years , necessitating a comprehensive survey to offer a bird’s-eye view of the current state of spatial crowdsourcing literature . In this article , we discuss the spatial crowdsourcing infrastructure and identify the fundamental differences between spatial and conventional crowdsourcing . Furthermore , we provide a comprehensive view of the existing literature by introducing a taxonomy , elucidate the issues/challenges faced by different components of spatial crowdsourcing , and suggest potential research directions for the future .
In most cases , existing approaches usually take the matching algorithm as a fundamental step to solve this problem which aims to maximize the number of completed tasks . However , in the present of many spatial crowdsourcing platforms , how to assign the tasks at high efficiency and make a relatively fair schedule for multiple workers is a new challenge . In this paper , we study the problem of load balancing based task scheduling for multiple workers .
With the rapid development of Mobile Internet and Online To Offline ( O2O ) marketing model , various spatial crowdsourcing platforms , such as Gigwalk and Gmission , are getting popular . Most existing studies assume that spatial crowdsourced tasks are simple and trivial .
Spatial Crowdsourcing Platforms ( SCP ) are systems that act as brokers that help people requesting location dependent tasks to find suitable workforce to accomplish it . This kind of system has been source of much interest for both academy and industry .
This list has been identified and validated by an analysis of scientific related works and several commercial SCPs . Based on this list , we also propose a generic architecture for SCPs that is meant to help the understanding of SCPs and the design of better spatial crowdsourcing solutions .
Spatial crowdsourcing services , such as Uber and Grabhub , become popular recently . Task assignment plays an important role in offering high-quality services .
Under the premise that an intrinsic motivation of contestants on crowdsourcing platforms is to solve challenging problems , a theoretical model of the relationships between the two measures and the contest outcome , mediated by the quality of contestants , is developed . Data from a popular software crowdsourcing platform , TopCoder , is used to validate hypotheses on the direct , interaction , and nonlinear impacts of the quality level and quality ambiguity of a prior phase contest ’s winning solution on the outcome of the focal phase contest . Results show an inverted U-shaped relationship between the quality ambiguity of a prior phase contest ’s winning solution and the quality of the contestants in the focal phase contest , highlighting a sweet spot for quality ambiguity .
how to leverage Git 's powerful yet underused advanced features . The last Web extra at http://youtu.be/SK6TBI1bNLI is a video of Thomas Baden , Chief Information Officer , State of Minnesota , Department of Human Services , describing the experience of working on the TopCoder Platform and with the members of the TopCoder Community .
First , we derive a set of key concerns from the crowdsourcing literature as an initial analytical framework for an exploratory case study in a Fortune 500 company . We complement the case study findings with an analysis of 13,602 crowdsourcing competitions over a ten-year period on the very popular Topcoder crowdsourcing platform . Drawing from our empirical findings and the crowdsourcing literature , we propose a theoretical model of crowd interest and actual participation in crowdsourcing competitions .
Research to date on crowdsourcing software development has tended to focus on the ' crowd ' or the technical platform , with little research from the perspective of the customer who is seeking to leverage the crowdsourcing development model . The findings from an in-depth case study of crowd-sourcing software development in a Fortune 500 company are augmented with an analysis of over 13,000 crowdsourcing competitions over a ten-year period on the Topcoder crowdsourcing platform , one of the most popular platforms for software development , are drawn on to evaluate the effectiveness of crowdsourcing in a software development context
We justify a definition of performance using the logarithm of a player 's rank . We apply the definition to rating TopCoder SRM . We improve the accuracy , guided by experimental results .
Presenting an algorithm to construct worker competition network and extract its network metrics features . The proposed framework was evaluated on 4,872 software crowdsourcing tasks , extracted from TopCoder platform , using five machine learners , compared with in-house TopCoder predictor . Results : 1 ) Workers reliability , links in the description , number of registered workers , number of required technologies , and task-workers network modularity are the most influencing factors for predicting crowdsourcing failure ; 2 ) The top-three learners for task failure are Naïve Bayes , Random Forest , and StackingC , with precision above 98.8 % , recall above 81.2 % , and F-measure above 91.2 % ; and 3 ) The proposed best learners significantly outperform the two baseline models in our evaluation .
Some examples include problems that require more than one data structure and/or algorithm , problems that require modifications to standard data structures and/or algorithms , and problems that require attention to efficiency both in time and space . The Topcoder competitive coding platform can be an excellent resource to provide these examples and can be used in several ways in the classroom . In this paper , we detail our experience with Topcoder in several classes and from several perspectives .
The prize of each task may be determined in terms of the expected duration of the task , its complexity , its quality , the required effort to complete , and the online reputation of the client . TopCoder awarding mechanism , performance-based , game theory-based , deadline driven or revenue sharing models are several approaches for determining task prizes in crowdsourced software development ( CSD ) . In this paper , Putnam model , a well-known software effort estimation model , is proposed to be applicable as a prize determination approach in crowdsourcing .
To develop the web-based tool , the HH&P utilized the NASA Tournament Lab ( NTL ) to develop the software with TopCoder under an existing contract . In this way , the HH&P is using one new tool ( the NTL and TopCoder ) to develop the next generation tool , the SMG . The beta-phase of the SMG is planed for release in the spring of 2014 and results of the beta-phase testing will be available for the IAC meeting in September .
The paper then defines a game theory model to analyze the primary factors in these minmax competition rules that affect the nature of participation as well as the software quality . Finally , using the proposed evaluation framework , this paper illustrates two crowdsourcing processes , Harvard-TopCoder and AppStori . The framework demonstrates the sharp contrasts between both crowdsourcing processes as participants will have drastic behaviors in engaging these two projects .
Such sharing , however , is at tension with the competitive nature of crowd contests . To understand this tension , this study investigates community-level sharing of code on Kaggle , a contest platform for predictive modeling . Analyzing data on 25 contests in 2015 and 2016 , we find that 10 % of users shared code during contests , that participants doing medium well in the contest were the most likely to share code , and that sharing code improved individual , but not collective performance .
The topic of the 2014 ALTA shared task was to identify location information in tweets . As in past competitions , we used Kaggle in Class as the framework for submission , evaluation and communication with the participants . In this paper we describe the details of the shared task , evaluation method , and results of the participating systems .
This paper describes Task 2 of the DCASE 2018 Challenge , titled " General-purpose audio tagging of Freesound content with AudioSet labels " . This task was hosted on the Kaggle platform as " Freesound General-Purpose Audio Tagging Challenge " . The goal of the task is to build an audio tagging system that can recognize the category of an audio clip from a subset of 41 diverse categories drawn from the AudioSet Ontology .
Coupled with transfer learning and hyper-parameter tuning , we adopt AlexNet , VggNet , GoogleNet , ResNet , and analyze how well these models do with the DR image classification . We employ publicly available Kaggle platform for training these models . The best classification accuracy is 95.68 % and the results have demonstrated the better accuracy of CNNs and transfer learning on DR image classification .
A solution to the Kaggle competition : Human Protein Atlas Image Classification ( HPA ) . Using microscopic images of cells provided by the Human Protein Atlas , convolutional neural networks , CNNs , were used to analyze and predict the location of protein patterns . Challenges included working with an unbalanced dataset , finding a correct learning rate , and choosing a correct architecture to solve the problem . Background Kaggle is an online platform for predictive modelling and analytics competitions , that hosts the HPA competition . In this competition , competitors try to build models to classify mixed patterns of proteins found in microscopic images .
This document describes the models and software used to build our solution . Reproducing the results for our solution can be done by running the scripts in-cluded in the Kaggle package
et al. ( 2011 ) . We show that our automated approach performs better than the best performing Kaggle entry and generalizes as a method to the Mohler dataset .
Hence , University of Texas MD Anderson Cancer Center organized two public radiomics challenges in head and neck radiation oncology domain . This was done in conjunction with MICCAI 2016 satellite symposium using Kaggle-in-Class , a machine-learning and predictive analytics platform . We drew on clinical data matched to radiomics data derived from diagnostic contrast-enhanced computed tomography ( CECT ) images in a dataset of 315 patients with oropharyngeal cancer .
LL-CNN was compared with 2D-UNet , 3D-UNet , a single-task CNN ( ST-CNN ) , and a pure multitask CNN ( MT-CNN ) . Training , validation , and testing followed Kaggle competition rules , where 160 patients were used for training , 20 were used for internal validation , and 20 in a separate test set were used to report final prediction accuracies . RESULTS On average contours generated with LL-CNN had higher Dice coefficients and lower RMSE than 2D-UNet , 3D-Unet , ST- CNN , and MT-CNN .
For the application , we worked with 3168 recorded voices , for which the results of an acoustic analysis are presented , registering variables such as frequency , spectrum , modulation , among others , seeking to obtain a pattern of identification and classification according to gender through a voice identifier . The data record used is in open access and can be downloaded from the Kaggle web platform via r. For the development of the algorithm ’s model , the statistical program R was used . Additionally , applications were made with Python by the development of classification algorithms .
Using five-fold cross-validation , we train a 3D attention-based network under two scenarios : ( 1 ) single-task learning with only classification , and ( 2 ) multi-task learning with both classification and regression . The single-task learning leads to a higher AUC compared with the Kaggle challenge winner pre-trained model ( 0.878 v. 0.856 ) , and multi-task learning significantly improves the single-task one ( AUC 0.895 , p<0.01 , McNemar test ) . In summary , the image-based predicted CFPT can be used in follow-up year lung cancer prediction and data assessment .
Study findings indicate that online crowdsourcing , a process that allows access to many participants to complete specific tasks , is a potentially viable resource for evaluation designs where access to comparison groups , large budgets , and/or time are limited . The article highlights the strengths and limitations of the online crowdsourcing approach and describes ways that it could potentially be used in evaluation practice .
Mobile and social technologies are providing new opportunities to document , characterize , and gather impressions of urban environments . In this article , we present a study that examines urban perceptions of three cities in central Mexico ; the study integrates a mobile crowdsourcing framework to collect geo-localized images of urban environments by a local youth community , an online crowdsourcing platform to gather impressions of urban environments along 12 physical and psychological dimensions , and a deep learning framework to automatically infer human impressions of outdoor urban scenes . Our study resulted in a collection of 7,000 geo-localized images containing outdoor scenes and views of each city ’s built environment , including touristic , historical , and residential neighborhoods , and 144,000 individual judgments from Amazon Mechanical Turk .
ABSTRACT The objective of this study is to explore the factors that are keys for an idea to be implemented on an online crowdsourcing platform . A data set of 320 implemented ideas from My Starbucks Idea – an online crowdsourcing platform – has been analyzed . We find that only one out of 500 users ’ submitted ideas are selected for implementation .
The objective of this study is to explore the factors that are keys for an idea to be implemented on an online crowdsourcing platform . A data set of 320 implemented ideas from My Starbucks Idea – an online crowdsourcing platform – has been analyzed . We find that only one out of 500 users ’ submitted ideas are selected for implementation .
A growing interest in using crowdsourcing to collect and annotate datasets has been observed in recent years . In this paper , we have implemented a toolbox to easily develop online crowdsourcing tools to build a corpus of virtual agent ’s non-verbal behaviors directly rated by users . We present two developed online crowdsourcing tools that have been used to construct a repertoire of virtual smiles and to define virtual agents ’ non-verbal behaviors associated to social attitudes .
In this paper we investigate to what extent online crowdsourcing can be used for the specific case of safety perceptions about people . We built an image – based online crowdsourcing platform , collected safety perception ratings and background information from more than 700 people and used them to quantitatively evaluate established theories based on qualitative research . On one hand , we show in this paper that online , image-based crowdsourcing can be used to gather perceptions about people too , not just architecture , confirming established theories based on qualitative work .
The successful execution of knowledge crowdsourcing ( KC ) tasks requires contributors to possess knowledge or mastery in a specific domain . The need for expert contributors limits the capacity of online crowdsourcing marketplaces to cope with KC tasks . While online social platforms emerge as a viable alternative source of expert contributors , how to successfully invite them remains an open research question .
While online Crowdsourcing marketplaces provide a powerful avenue for facilitating new forms of information- driven micro-labor , their practical value is significantly reduced by worker “ spam ” and employer fraud . We hypothesize anonymity of parties is a major source of these problems , and we thus propose a human-centric solution : encourage employers and workers to voluntarily de- anonymize in order to reap a potential benefit of more productive and profitable labor interactions .
Mobile crowdsourcing markets ( e.g. , Gigwalk and TaskRabbit ) offer crowdworkers tasks situated in the physical world ( e.g. , checking street signs , running household errands ) . The geographic nature of these tasks distinguishes these markets from online crowdsourcing markets and raises new , fundamental questions . We carried out a controlled study in the Chicago metropolitan area aimed at addressing two key questions : ( 1 ) What geographic factors influence whether a crowdworker will be willing to do a task ?
To do so we carried out two studies where participants performed 3 communication tasks , solo in study one and dyadic in study 2 , and were recorded on video both with and without robot mediation . Judges recruited using online crowdsourcing services then made personality judgements of those in the video clips . We observed that judges were able to make internally consistent trait judgements in both communication conditions .
Several software crowdsourcing models performed on typical platforms like Topcode , uTest are established and their organization and coordination are discussed . We have developed a service-based multi-agent system platform called AutoService that provides some fundamental capabilities like autonomy , monitoring , flexible interaction and organization , and can serve as an infrastructure to support software crowdsourcing models and tackle its challenges . A software crowdsourcing prototype is developed and some scenarios are exemplified to illustrate our approach .
With the increasing role of software crowdsourcing , a new working architecture for software development , in global software development ( GSD ) settings , has emerged challenges regarding sustainable software development and software sustainability , which are still to be elucidated . Software standards and models exist to assist co-located and global software development ; however , seismic changes in the working architecture of software development as software crowdsourcing are highlighting a risk of software failure . Recently , statistics showed that only 30 % of projects are successful , whereas 50 % encounter challenges and the remaining 20 % are absolute failures that implicate a challenge toward software success and its sustainability .
Thus , a lack of any guideline toward sustainable software crowdsourcing is highlighted as one of the limitations in the software standards . This paper contributes to the GSD industry in two folds , first by laying down the importance and theoretical association on and around sustainable crowdsourcing , and second by highlighting that the IEEE standards , inadequacy toward the suppressing phenomena of sustainability in software crowdsourcing , if addressed adequately , can minimize the ripple effect toward software failure .
However , existing rating systems are facing challenges such as providing limited information regarding worker 's abilities as well as potential threats from workers ' gaming or cheating the systems . To develop better understanding of worker performance in software crowdsourcing , this paper reports an empirical study at TopCoder , one of the primary software crowdsourcing platforms . We aim at investigating the following questions : How diverse are crowd workers in terms of skill and experience ?
The key idea of CrowDevBot is to : ( 1 ) combine a rulebased method and an SVM-NaiveBayes-C4.5 integrated learning method to discover users ’ intention ; ( 2 ) employ an integrated CRF ( conditional random field ) method with novel features to improve the performance of slot filling ; and ( 3 ) leverage a software service knowledge base to unify entity names and predefine the key slots of user query . We implement CrowDevBot and integrate it into JointForce , an IT software crowdsourcing platform in China . To the best of our knowledge , this is the first time that a task-oriented conversational bot is practically used in software crowdsourcing platform(s ) .
more relevant context , i.e. a set of similar tasks identified based on Topic modeling and we evaluate 7 predictive models . The proposed approach is evaluated on 450 software tasks extracted from TopCoder , the largest software crowdsourcing platform . The results show that : 1 ) the proposed models can be used at early crowdsourcing planning phase , when information on traditional metrics are not available ; 2 ) the best model achieves 65 % in accuracy measure ; 3 ) the local model exhibits 27 % increases superior to the global model .
The Gene Wiki has created over 10,000 Wikipedia articles , and aims to provide one for every notable human gene [ 10 ] . The crowdsourcing platform InnoCentive has reportedly facilitated solutions to roughly half of the thousands of technical problems posed on the site , including many in life sciences such as the $ 1 million ALS Biomarker Prize [ 11 ] . Other examples include prizes ( X-Prize [ 12 ] ) , scientific games ( FoldIt [ 13 ] ) , and licensing schemes inspired by open-source software ( BIOS [ 14 ] ) . Collaborative R&D approaches vary in openness [ 15 ] . In some approaches , the R&D process and outputs are open to all — for example , open-science projects like the Gene Wiki described above . In other approaches which demonstrate what might be called controlled collaboration , there are strong controls on who contributes and benefits — for example , computational platforms like Collaborative Drug Discovery or InnoCentive that support both commercial and nonprofit research [ 9 ] , [ 11 ] . Collaborative approaches can unleash innovation from unforeseen sources , as with crowdsourcing health technologies [11]–[13 ] , [ 16 ] .
In fact , spatial crowdsourcing has stimulated a series of recent industrial successes including Citizen Sensing ( Waze ) , P2P ride-sharing ( Uber ) and Real-time Online-To-Offline ( O2O ) services ( Instacart and Postmates ) . In this tutorial , we review the paradigm shift from web-based crowdsourcing to spatial crowdsourcing . We dive deep into the challenges and techniques brought by the unique spatio-temporal characteristics of spatial crowdsourcing .
Although some of them utilize this approach , they use one of the web-based crowdsourcing frameworks with paid crowdworkers . In this paper we decided to present our own developed web-based crowdsourcing framework and use it as unpaid approach for evaluating two well-known compression standards - H.264 and H.265 and for comparing the results getting from testing via this crowdsourcing framework with results which we got from laboratory testing . These results showed that the subjective quality testing via unpaid crowdsourcing approach can be used as a replacement to the subjective laboratory testing , especially in cases where not very sophisticated evaluation is required .
Agora aims to be an " operating system " for social media applications gathering data . Furthermore , the Web-based crowdsourcing platform metropolitalia has been established for hosting two social media applications based upon Agora : Mercato Linguistico and Poker Parole . These applications have been conceived as part of this thesis for gathering complementary data and meta-data on Italian language varieties .
Our proposed solution provides a collaborative platform to acquire assessments from a larger group of experts , towards forming a collective insight and minimizing overlooking of defects . Our first-of-its-kind web-based crowdsourcing platform can be leveraged for 3D visualization of microscopy data without imposing any computational load on the users , as well as collaborative analysis by collecting information from each user . Further , the collected information is compiled in a data bank , which serves as a valuable source for developing quantified measures and for training automated defect classification algorithms .
The requirements of enterprise crowdsourcing platforms are compiled based on the literature and a qualitative study , providing a conceptual design of task assignment strategies . The demands of workers and their perception of task similarity on public crowdsourcing platforms are identified , leading to the design and implementation of additional methods to determine the similarity of micro-tasks . The textual descriptions of micro-tasks , projects , and job postings are analyzed in order to provide innovative methods for task recommendation in these domains .
The 5C categorization is based on the action s enabled by the social media tools : communicating , collaborating , connecting , completing , and combining . In order to demonstrate how the 5C categorization can be used as a framework to assess social media tools we conducted an empirical study of social media based crowdsourcing platforms in business-to-business setting . The 5C categorization can be utilized in planning a social media strategy as it forwards the understanding of which tools are suitable for certain purpose and provides a scheme to evaluate and compare different social media tools and applications .
Crowdsourcing is a useful approach for organizing a database based on collaborative works of people . We recruited trial users to test our system via a web-based crowdsourcing service provider in Japan . However , we found that improvement of the system is important for maintaining the motivation of users in order to continue the collection of sounds .
Such combination requires an understanding of where the algorithms fail , which we gather using a novel coarse scribble video annotation task . We evaluate our ensemble methods , discuss our design choices for them , and make our web-based crowdsourcing tools and results publicly available .
In this paper we specifically focus on the impact of online citizen science ( OCS ) participation in the science education of primary school age children in New Zealand . Methods : We use four exploratory cases within a broader research project to examine the nature and impact of embedding OCS projects that use web based online crowdsourcing and collaboration tools within classroom environments of primary school science learners . Results & Discussion : Our findings provide insights into primary school teachers ' perception of OCS .
This work examines to which extend , " trapping question " or " outliers detection " assure reliable results to the detriment of , overloading task content with stimuli that are not of interest for the researcher , or by discarding data points that might be the true opinion of a worker . To this end , a speech quality assessment study have been conducted in a web crowdsourcing platform , following the ITU-T Rec . P.800 .
To reduce the time cost , we adopt split techniques and design Simple Split(SS ) and Dynamic Split(DS ) algorithms . We verify the effectiveness of our solutions by conducting crowdsourcing simulation experiments and experiments on a crowdsourcing platform namely gMission .
The third paper , titled Toward an Understanding of Participants ’ sustained participation in Crowdsourcing Contests by Xuan Wang , Hanieh Javadi Khasraghi , and Helmut Schneider , addresses a little known issue regarding the factors that influence individuals ’ continued participation in crowdsourcing contests . The authors conduct an empirical study using data from an online crowdsourcing contest platform , Kaggle , which delivers data science and machine learning solutions and models to its clients . The analysis finds that both collaborative and competitive components such as tenure , performance , price amount , number of competitors , and
Resource Allocation and Alignment Effects . Abstract Advances in information technology bring changes to the nature of work by facilitating companies to go beyond the wisdom of their workforce and tap into the “ wisdom of the crowd ” via online crowdsourcing contests . In these contests , active and motivated individuals collaborate in the form of self-organized teams that compete for rewards .
Recent advances in information technology bring significant changes to the nature of work by allowing companies to tap “ wisdom of the crowd ” beyond their own workforce . In particular , organizations increasingly seek solutions to their business problems using online crowdsourcing contests that are open to the public . In these contests , self&organized vi rtual teams compete for monetary reward .
However , there are some barriers preventing them from adopting crowdsourcing into their product design and development ( PDD ) practice . In this paper , we provide a literature review of key crowdsourcing technologies including crowdsourcing platforms and tools , crowdsourcing frameworks , and techniques in terms of open call generation , rewarding , crowd qualification for working , organization structure of crowds , solution evaluation , workflow and quality control and indicate the challenges of integrating crowdsourcing with a PDD process . We also explore the necessary techniques and tools to support the crowdsourcing PDD process .
An exploratory study on perception of Indian crowd towards crowdsourcing software development Crowdsourcing metadata quality: A survey of potential tools and use cases Improving the credibility of unreliable information through static images and data mining: an experimental study to identify floods Enabling Human-Centric Smart Cities: Crowdsourcing-Based Practice in China Predicting the quality of contests on crowdsourcing-based software development platforms: student research abstract Modeling Method of Resource Combination Optimization for Crowdsourcing Product Development Based on Cloud Citizen OBservatory WEB (COBWEB): A Generic Infrastructure Platform to Facilitate the Collection of Citizen Science data for Environmental Monitoring Crowdsourcing IT Work: A Three-Fold Perspective from the Workers, Buyers, and Platform Providers Learning to Count Leaves of Plants Krik: First Steps into Crowdsourcing POS tags for Kréyòl Gwadloupéyen Crowdsourcing service requirement oriented requirement pattern elicitation method A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces A Survey of Spatial Crowdsourcing Efficient and Load Balancing Strategy for Task Scheduling in Spatial Crowdsourcing Top-k Team Recommendation in Spatial Crowdsourcing A generic architecture for spatial crowdsourcing Minimizing Maximum Delay of Task Assignment in Spatial Crowdsourcing Examining Multi-Phased Online Crowdsourcing Contests Using Quality Level and Quality Ambiguity Social Networking Meets Software Development: Perspectives from GitHub, MSDN, Stack Exchange, and TopCoder Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective Crowdsourcing Software Development: Silver Bullet or Lead Balloon An Elo-based rating system for TopCoder SRM Failure Prediction in Crowdsourced Software Development Using topcoder in introductory data structures and algorithms A Prize Determination Approach for Crowdsourced Software Development Novel Problem Solving - The NASA Solution Mechanism Guide An evaluation framework for software crowdsourcing To Share, or Not to Share? Overview of the 2014 ALTA Shared Task: Identifying Expressions of Locations in Tweets General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline Deep convolutional neural networks for diabetic retinopathy detection by image classification Convolutional Neural Networks for Protein Image Classification Kaggle LSHTC4 Winning Solution Identifying Patterns For Short Answer Scoring Using Graph-based Lexico-Semantic Text Matching Machine Learning Applications in Head and Neck Radiation Oncology: Lessons From Open-Source Radiomics Challenges A convolutional neural network algorithm for automatic segmentation of head and neck organs at risk using deep lifelong learning. Bosques aleatorios como extensión de los árboles de clasificación con los programas R y Python Deep Multi-task Prediction of Lung Cancer and Cancer-free Progression from Censored Heterogenous Clinical Imaging Finding a Comparison Group Looking South Generating Ideas on Online Platforms: A Case Study of “My Starbucks Idea” User Ideation on Online Platforms: A Case Study of My Starbucks Idea A crowdsourcing toolbox for a user-perception based design of social virtual actors Crowdsourcing Safety Perceptions of People: Opportunities and Limitations On the Invitation of Expert Contributors from Online Communities for Knowledge Crowdsourcing Tasks Enabling trust in crowd labor relations through identity sharing Avoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets Personality Perception of Robot Avatar Teleoperators in Solo and Dyadic Tasks Multi-Agent System Approach for Modeling and Supporting Software Crowdsourcing Investigating Software Standards: A Lens of Sustainability for Software Crowdsourcing Leveraging Crowdsourcing for Team Elasticity: An Empirical Evaluation at TopCoder CrowDevBot: A Task-Oriented Conversational Bot for Software Crowdsourcing Platform (S) Context-Centric Pricing: Early Pricing Models for Software Crowdsourcing Tasks Ten Simple Rules for Cultivating Open Science and Collaborative R&D Spatial Crowdsourcing: Challenges, Techniques, and Applications QoE on H.264 and H.265: Crowdsourcing versus Laboratory Testing Crowdsourcing for linguistic field research and e-learning A novel crowdsourcing platform for microelectronics counterfeit defect detection Task Recommendation in Crowdsourcing Platforms The 5C categorization of social media tools Development of a Mobile Application for Crowdsourcing the Data Collection of Environmental Sounds Crowd-Guided Ensembles: How Can We Choreograph Crowd Workers for Video Segmentation? Using the Web for Science in the Classroom: Online Citizen Science Participation in Teaching and Learning Outliers Detection vs. Control Questions to Ensure Reliable Results in Crowdsourcing.: A Speech Quality Assessment Case Study Knowledge Base Enhancement via Data Facts and Crowdsourcing From the Special Issue Editors Task Division for Team Success in Crowdsourcing Contests: Resource Allocation and Alignment Effects Virtual Team Performances in Crowdsourcing Contests Key Crowdsourcing Technologies for Product Design and Development
d675d795a8911dfc1ff5cf8eff24b18e7b2364e4 bf1683e2a6dac44e3fb60beb7a554dc7ebcf0054 4f91123dceb8a0811e8ded44a8f2f766c39bccb2 9f46128596cab35d41b176e0b6879602b683f956 93b665c4cd79ba48be1daab389821620103fd7f7 70c3469300b0bbf234ec779a87cd8e49cd17f9f3 d838d43a3c48c564a2b0555fbc57328fb01f2187 9b4895751fc70745ccd25f4d196dde533fe8d87b ff43df0d4d405fc02c6a133799a858243e087ba6 a0eedd8a915ad8317cc1d94c0a932109d9f05a05 eaf637dfdc91a2906328b78327f88666fdb48849 ab13e94f66ad7b4c0b6416df34e64f5eb0ecb466 ab86dfb48fb8b9158581adfb83ab99d698543fcf 81a1cbf539f4d9c355a709f8bba471e9ac146d77 1e8f65c9aedfee245a173a59de843f51312f2932 bf6f3498d887da7b9dea07a45ac500638a6ab624 310863e80ac622dffbbde4385b6240385dcbc181 2f9619a071f728bb8c55fa368cac026f3ebedfd6 40f5e5f73ecd5cf5cad4059cfda47362e0a9b427 a1d0cd8ca9bb823f9862c8b884eeae3b878597a1 2f603d0baed7ef3e833a3bb62f78f53e696f48ca 8bb944a0b6f2b2395673c6d154fb5d595e9fad2c 1e94475cbfc0494c05b37095e3a347d5e3ef855e cb2fc8bc4015511523e476ab1832b2dc8681049e add027251eff5c9f8908a59a06aef618afa87d30 39846627ac07d1ea7c7b7b8f725cc09217806a7e a142b3c969283594641532900fff55ca1b50eacb 082113901825964aecc2386cb4fcd3e095670cac 4884e0c11257bc99b31a32fb6e4568725ea79fd4 82c8651acf01e2ba4530f541d173805870a62c79 e4ab292177b2b410e2d99dc83191b9cfea4df707 38c4ea90223c6905aba4274b9552ff9abe59de38 9a3cdd8b6c3a9d5d2fd90d18a5463e3f0741511a 71236c2ea1b4dee3ffe2d87ef762e3c606530d84 4656fa1257d009fe756a3d6e29de848dbdaeb689 92822c08c14141bf36e650957372733fb90d4055 2a8ab85fa9acdaefa43c43763aa9b3cff5016a11 2ecfe5b6d98a5ce5706495f05d16c350fde76355 3ebef352bc305555520d5a60d7cf44e2d32eb69d 0c7adc889b309ee72ea731fde51c7fdaf40385c4 880ab85b48caffda2f31315c489081056d15f763 fae58f145ce1b91508dc4b1d3fa8d9963c34e93c 7ce17c9ded1b66fd008e5d1d4eaecd42990831d7 841b02870a64fd80d0e578e2078e0874a3cda7ec 50d97a84dce0ac0fe102eaa7065a58bddc3b4a04 8594ad5beee519dc522ac281f1a4959b6b3eb084 b92c1786b2c656c19d54235f637f47850c7ff10f 23fe3b3dcd04ebc8fd556f9864325c8a5b772281 85d3bdcb583712b512b0e9b75a724151d43d4b68 f81d72bdb48f7e946e984b9fea9e8d3041a222ba c536437b2936f94076a0b05e25f7d432ac6e6796 ef8896a405b7fcdda05897f523a0b66c0b976be1 1e5f096305b95cc754e42844cdedd52c339d126d 92eb95cfca38c97e4a7cac2813191a48a4c0d127 24f0a580ee78c1e8992ec0947d8ea02f3b92caca abd00e4dd4f26a59585ac6e8e7bb34e434ab0197 ae49d60ebb1f1011c7f4181813392c43bb7f583c 56ad74e502ed4545367f7123d2bd025f7f09dd59 d4cd652c7efb9f2beb86acfe7ca43fa91217d1cb 9f35c8372029f1060a85ed30690854ff58d6bcda 030ab7710a1307cfa2159686dabd07e2b239fb10 23a7dae3f232a8ea023c2b510ca095b3c520d230 c5234f40567000aba7b8168e3923c035756ad769 e3564cc621b4612af9e6fcb3164dee7d3f68d795 290345422717fd9b042d601c3600e7d4cc9575eb 87105f1d03f6aef577756200ef31a90d6522d6b9 ab5c4625267d4b64f01e89007f6cc1be3e034ef6 41b3c3284915074ef3a90d4e9bea89511110e795
["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Geography"] ["Computer Science","Business"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Business","Computer Science"] ["Computer Science"] ["Computer Science","Engineering","Mathematics"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Medicine"] ["Medicine","Physics"] ["Computer Science"] ["Computer Science","Medicine"] ["Psychology"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Business","Computer Science"] ["Computer Science","Geography"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Medicine","Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Engineering"] ["Computer Science","Engineering"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science"] ["Computer Science","Mathematics"]
2017 2016 2016 2018 2017 2017 2016 2015 2018 2018 2020 2018 2018 2019 2016 2016 2016 2019 2017 2013 2019 2018 2019 2017 2017 2018 2014 2013 2017 2014 2018 2018 2019 2014 2015 2018 2019 2017 2019 2013 2018 2015 2015 2013 2015 2016 2011 2015 2017 2015 2018 2017 2019 2017 2013 2017 2020 2014 2018 2019 2016 2014 2018 2019 2018 2018 2020 2015 2014 2019
ISCRAM SAC CPR ACL ArXiv WAIM Workshops WAIM (1) RCIS ICDE AIRE@RE APSEC EECS ALTA BEA@NAACL-HLT SocInfo ICWE ASIST CSCW ICSE-SEIP SEKE PROMISE MindTrek HCI (12) CHI WebSci WWW (Companion Volume) ICDE AMCIS
https://api.semanticscholar.org/d675d795a8911dfc1ff5cf8eff24b18e7b2364e4 https://api.semanticscholar.org/bf1683e2a6dac44e3fb60beb7a554dc7ebcf0054 https://api.semanticscholar.org/4f91123dceb8a0811e8ded44a8f2f766c39bccb2 https://api.semanticscholar.org/9f46128596cab35d41b176e0b6879602b683f956 https://api.semanticscholar.org/93b665c4cd79ba48be1daab389821620103fd7f7 https://api.semanticscholar.org/70c3469300b0bbf234ec779a87cd8e49cd17f9f3 https://api.semanticscholar.org/d838d43a3c48c564a2b0555fbc57328fb01f2187 https://api.semanticscholar.org/9b4895751fc70745ccd25f4d196dde533fe8d87b https://api.semanticscholar.org/ff43df0d4d405fc02c6a133799a858243e087ba6 https://api.semanticscholar.org/a0eedd8a915ad8317cc1d94c0a932109d9f05a05 https://api.semanticscholar.org/eaf637dfdc91a2906328b78327f88666fdb48849 https://www.semanticscholar.org/paper/0a78873e41615798d09391d9f40d41666b8c9beb https://www.semanticscholar.org/paper/15c10b24ef645d83ff4059affd86945c33e00328 https://api.semanticscholar.org/ab13e94f66ad7b4c0b6416df34e64f5eb0ecb466 https://api.semanticscholar.org/ab86dfb48fb8b9158581adfb83ab99d698543fcf https://api.semanticscholar.org/81a1cbf539f4d9c355a709f8bba471e9ac146d77 https://api.semanticscholar.org/1e8f65c9aedfee245a173a59de843f51312f2932 https://api.semanticscholar.org/bf6f3498d887da7b9dea07a45ac500638a6ab624 https://api.semanticscholar.org/310863e80ac622dffbbde4385b6240385dcbc181 https://api.semanticscholar.org/2f9619a071f728bb8c55fa368cac026f3ebedfd6 https://api.semanticscholar.org/40f5e5f73ecd5cf5cad4059cfda47362e0a9b427 https://api.semanticscholar.org/a1d0cd8ca9bb823f9862c8b884eeae3b878597a1 https://api.semanticscholar.org/2f603d0baed7ef3e833a3bb62f78f53e696f48ca https://api.semanticscholar.org/8bb944a0b6f2b2395673c6d154fb5d595e9fad2c https://api.semanticscholar.org/1e94475cbfc0494c05b37095e3a347d5e3ef855e https://api.semanticscholar.org/cb2fc8bc4015511523e476ab1832b2dc8681049e https://api.semanticscholar.org/add027251eff5c9f8908a59a06aef618afa87d30 https://api.semanticscholar.org/39846627ac07d1ea7c7b7b8f725cc09217806a7e https://api.semanticscholar.org/a142b3c969283594641532900fff55ca1b50eacb https://api.semanticscholar.org/082113901825964aecc2386cb4fcd3e095670cac https://api.semanticscholar.org/4884e0c11257bc99b31a32fb6e4568725ea79fd4 https://api.semanticscholar.org/82c8651acf01e2ba4530f541d173805870a62c79 https://api.semanticscholar.org/e4ab292177b2b410e2d99dc83191b9cfea4df707 https://api.semanticscholar.org/38c4ea90223c6905aba4274b9552ff9abe59de38 https://api.semanticscholar.org/9a3cdd8b6c3a9d5d2fd90d18a5463e3f0741511a https://api.semanticscholar.org/71236c2ea1b4dee3ffe2d87ef762e3c606530d84 https://api.semanticscholar.org/4656fa1257d009fe756a3d6e29de848dbdaeb689 https://api.semanticscholar.org/92822c08c14141bf36e650957372733fb90d4055 https://api.semanticscholar.org/2a8ab85fa9acdaefa43c43763aa9b3cff5016a11 https://api.semanticscholar.org/2ecfe5b6d98a5ce5706495f05d16c350fde76355 https://api.semanticscholar.org/3ebef352bc305555520d5a60d7cf44e2d32eb69d https://api.semanticscholar.org/0c7adc889b309ee72ea731fde51c7fdaf40385c4 https://api.semanticscholar.org/880ab85b48caffda2f31315c489081056d15f763 https://api.semanticscholar.org/fae58f145ce1b91508dc4b1d3fa8d9963c34e93c https://api.semanticscholar.org/7ce17c9ded1b66fd008e5d1d4eaecd42990831d7 https://api.semanticscholar.org/841b02870a64fd80d0e578e2078e0874a3cda7ec https://api.semanticscholar.org/50d97a84dce0ac0fe102eaa7065a58bddc3b4a04 https://api.semanticscholar.org/8594ad5beee519dc522ac281f1a4959b6b3eb084 https://api.semanticscholar.org/b92c1786b2c656c19d54235f637f47850c7ff10f https://api.semanticscholar.org/23fe3b3dcd04ebc8fd556f9864325c8a5b772281 https://api.semanticscholar.org/85d3bdcb583712b512b0e9b75a724151d43d4b68 https://api.semanticscholar.org/f81d72bdb48f7e946e984b9fea9e8d3041a222ba https://api.semanticscholar.org/c536437b2936f94076a0b05e25f7d432ac6e6796 https://api.semanticscholar.org/ef8896a405b7fcdda05897f523a0b66c0b976be1 https://api.semanticscholar.org/1e5f096305b95cc754e42844cdedd52c339d126d https://api.semanticscholar.org/92eb95cfca38c97e4a7cac2813191a48a4c0d127 https://api.semanticscholar.org/24f0a580ee78c1e8992ec0947d8ea02f3b92caca https://api.semanticscholar.org/abd00e4dd4f26a59585ac6e8e7bb34e434ab0197 https://api.semanticscholar.org/ae49d60ebb1f1011c7f4181813392c43bb7f583c https://api.semanticscholar.org/56ad74e502ed4545367f7123d2bd025f7f09dd59 https://api.semanticscholar.org/d4cd652c7efb9f2beb86acfe7ca43fa91217d1cb https://api.semanticscholar.org/9f35c8372029f1060a85ed30690854ff58d6bcda https://api.semanticscholar.org/030ab7710a1307cfa2159686dabd07e2b239fb10 https://api.semanticscholar.org/23a7dae3f232a8ea023c2b510ca095b3c520d230 https://api.semanticscholar.org/c5234f40567000aba7b8168e3923c035756ad769 https://api.semanticscholar.org/e3564cc621b4612af9e6fcb3164dee7d3f68d795 https://api.semanticscholar.org/290345422717fd9b042d601c3600e7d4cc9575eb https://api.semanticscholar.org/87105f1d03f6aef577756200ef31a90d6522d6b9 https://api.semanticscholar.org/ab5c4625267d4b64f01e89007f6cc1be3e034ef6 https://api.semanticscholar.org/41b3c3284915074ef3a90d4e9bea89511110e795