Developer Recommendation for Crowdsourced Software Development Tasks
|
|
- Ross Lewis
- 6 years ago
- Views:
Transcription
1 2015 IEEE Symposium on Service-Oriented System Engineering Developer Recommendation for Crowdsourced Software Development Tasks Ke Mao *,YeYang, Qing Wang, Yue Jia *, Mark Harman * * CREST Centre, University College London, Gower Street, London, WC1E 6BT, UK. School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, 07030, USA. Lab for Internet Software Technologies, Institute of Software, Chinese Academy of Sciences, Beijing, , China. k.mao@cs.ucl.ac.uk, ye.yang@stevens.edu, wq@itechs.iscas.ac.cn, yue.jia@ucl.ac.uk, m.harman@ucl.ac.uk Abstract Crowdsourced software development utilises an open call format to attract geographically distributed developers to accomplish various types of software development tasks. Although the open call format enables wide task accessibility, potential developers must choose from a dauntingly large set of task options (usually more than one hundred available tasks on TopCoder each day). Inappropriate developer-task matching may lower the quality of the software deliverables. In this paper, we employ content-based recommendation techniques to automatically match tasks and developers. The approach learns particular interests from registration history and mines winner history to favour appropriate developers. We measure the performance of our approach by defining accuracy and diversity metrics. We evaluate our recommendation approach by introducing 4 machine learners on 3,094 historical tasks from TopCoder. The evaluation results show that promising accuracy and diversity are achievable (accuracy from 50% to 71% and diversity from 40% to 52% when recommending reliable developers). We also provide advice extracted from our results to guide the crowdsourcing platform in building a recommender system in practice. I. INTRODUCTION In recent years, an increasing number of successful software companies have turned to employ decentralised software ecosystems such as open source communities and crowdsourcing to augment their software production [1]. Crowdsourced Software Development (CSD), which derives its concept from Crowdsourcing, utilises an open call format to attract online developers to accomplish various types of software development tasks such as architecture, component design, component development, testing and bug fixing. Jeff Howe [2] first coined the definition of crowdsourcing in Although crowdsourcing is not specially proposed for software engineering domains, its application in software development is growing. A crowdsourcing industry report from Massolution [3] indicates that the number of workers engaged in software development increased by 151% in the year This increase is even more dramatic than the increase in crowdsourced micro-tasks: Amazon Mechanical Turk (AMT) is one of the most popular marketplaces for crowdsourcing micro-tasks such as photo tagging, logo design and quick idea gathering. Crowdsourcing platforms that support software development include TopCoder, utest, GetACoder, elance, Guru, Freelancer, Tackcn, etc. Among them, TopCoder 1 is 1 TopCoder Website: the world s biggest competitive software development portal [4]. Its clients include Google, Microsoft, Facebook and AOL. Compared with traditional software development, TopCoder s crowdsourced development is claimed to exhibit the ability to deliver custom requested software assets with reduced defect rate, cost and time [5], [6]. Current crowdsourced software development practices usually use an open call format such as online competitions. Although this enables wide task accessibility and self-selection features for the crowd developers, potential developers must choose from a dauntingly large set of task options. Chilton et al. [7] found that most workers usually view only a few recent tasks posted on the AMT micro-task crowdsourcing platform. Considering the uneven expertise and skill levels of the crowd developers, inappropriate developer-task matching may decrease the quality of the software deliverables. Specifically, the resulting consequences can be viewed from two different perspectives: 1) From the developer s perspective: Developers from around the world can select the types of tasks in which they would like to compete. However, under the situation that there are lots of simultaneously competitive tasks posted on crowdsourcing platforms such as TopCoder, it is quite a laborious and demanding work for the developers to choose which one is more suitable for them to undertake. Usually there are more than one hundred simultaneously active online tasks on TopCoder for a single day. For instance, the number is 222 on Dec. 12, 2014 as shown in Figure 1. 2) From the platform s perspective: It is a valuable but challenging task for the platform to seek best available developers: a key factor for delivering qualified software assets. In traditional outsourced software development, one of the most critical steps is vendor selection, which has a direct correlation with the quality of the outcome [8]. In crowdsourced software development, despite the fact that the vendor developers are attracted by an open call format rather than being selected, encouraging the right developers to participate still plays a crucial role in delivering qualified clients requested assets. Background and skill levels can vary significantly among crowd developers. Also, previous research showed that developer participation levels have an impact on CSD software quality [9]. Thus the platform may need to attract not only the right developers, but also as many participants as possible /15 $ IEEE DOI /SOSE
2 Fig. 1. Available tasks listed on TopCoder on Dec. 12, In order to tackle the challenges mentioned above, we employ content-based recommendation techniques to automatically match tasks to developers based on historical data and preference learning. The illustration of developer recommendation in the CSD context can be briefly shown in Figure 2. Although current CSD platforms such as TopCoder usually allow signing up for alerts about upcoming tasks [6], they do not provide personalised recommendation for such alerts. Our developer recommendation results for newly arriving tasks can be delivered to signed up developers by notification 2. The primary contribution of this study is that we introduce a framework named CrowdRex, which automatically recommends developers for newly arriving CSD tasks. The recommendation is for different purposes: our system learns from the winning history for recommending reliable developers and learns from registration history to suggest suitable participants for available tasks. Other contributions of the paper include multiple content features and their similarity measures as well as the formulation of the accuracy-diversity dilemma, for the Fig. 2. Developer recommendation utilising historical data. 2 According to a survey on TopCoder website ( 80.8% vote for as their preferred method to receive information about TopCoder events. CSD developer recommendation problem. The rest of this paper is organised as follows: Section II describes crowdsourced software development process, the work flow of open call tasks and participation incentives, as well as related recommendation techniques as background information. Section III introduces our approach to recommending developers for crowdsourced software development tasks. The evaluation of our work is presented in Section IV. Section V introduces related work. Finally Section VI concludes and points out directions for future work. II. BACKGROUND Prior to presenting our approach of developer recommendation, we outline the background relating to TopCoder s crowdsourced development process, open call tasks and related recommendation techniques for better understanding the construction of our recommender system. Other platforms may adopt similar processes but our evaluation was carried out using the historical data from TopCoder. A. Crowdsourced Software Development Since TopCoder is currently the world s largest CSD platform supporting a global crowd of more than 715,000 developers, it is based on this platform shall we introduce the typical CSD methodology. 1) Development Process: The crowdsourced software development process used by TopCoder generally follows a waterfall model, which is shown in Figure 3(a). Each of the phases is achieved by the open competition format, utilising TopCoder s global talent pool of developers. The development process commences with a requirement phase. During this phase, the project manager, who comes from the crowd or the TopCoder platform, is responsible for managing the following phases and communicating with the client companies to identify their project goals, task plan and estimated budget. Then the requirements specification is defined and passed as the input to the next phase. The subsequent architecture phase decomposes the application into 348
3 a set of components (TopCoder s crowdsourced software development practices component-based software engineering). The component design activity produces a full set of design documentations such as UML diagrams and component specifications. These specified design components are then implemented in the subsequent development phase. The component development activity may incorporate pre-built reusable components. The finished components are combined together in an assembly phase and are further certified by system-level testing activities. Assembly tasks require online developers to build the application by assembling winning components according to the architecture design. Finally, the fully functioning solution is deployed into the customer s quality assurance environment in the deployment phase. After a period of user acceptance testing, all developed assets are delivered to the client. For further maintenance activities, TopCoder provides Bug Hunt and Bug Race tasks for discovering and fixing bugs. 2) Open Call Tasks: Crowdsourcing utilises an open call format to attract online workers to make contributions to the posted tasks. TopCoder s open call format is in the form of a competition. Each crowdsourced development task is organised as an open contest. Every registered member satisfying the legal requirement can register for contests and submit their solutions. Usually the top two winners receive prize money as a reward. The typical open call task phases on TopCoder are divided into a series of task competitions. The duration of the whole process for a single task is usually 1-2 weeks. The process to run an open call task is illustrated in 3(b). To start with, a task categorised by its development type is posted on the website with information such as task descriptions, payment amount and time lines. The time lines include two important dates: registration deadline and submission deadline. All developers who are willing to participate should announce their decision publicly by registering the contest. This means online developers are allow to observe opponent developers information including historical performance and skill ratings. This registration phase usually lasts a few days. Registrants can obtain detailed documentation and are required to submit their solutions before the submission deadline. After the submission deadline, all submitted solutions are collected by the platform to be evaluated by peer review according pre-defined screening and review scorecards 3. The solutions that passed screening are scored (usually by three community experts from different perspective, e.g., performance on accuracy, stress and failure tests). Once the review process is finished, the developers are notified privately. If they are not satisfied with the results they have one chance to argue with the reviewers (called the appeal phase). Finally the revised scores are announced on the website and contestants are ranked by the average score given by all reviewers. The 1st place winner gets the full payment and the runner-up gets half of this amount. 3) Participation Incentives: The prize money is an important factor in motivating the crowd participation [11] [13], but not the only motivation. Statistics on TopCoder s historical assembly tasks show that only 13.6% developers had ever won a task, 23.4% developers had not ever won a task but registered at least 5 tasks and 21.8% developers had not ever won a task but kept active in participation for at least 180 days. There are multiple types of incentives for developers to undertake CSD tasks. According to TopCoder, these incentives include gaining skills, getting feedback, making friends, earning money having fun, getting peer recognition and getting sense of accomplishment [14]. Since usually only top two winners of the tasks can be rewarded with money, the developers often get nothing monetary for their effort. But through competing in the task competition, developers may find it exciting and their submissions are tested by screening and further reviewed by the experts of the community. In this way they are rewarded with fun, acknowledgement and experts feedback for improving their skills. According to a survey 4, 69.28% developers on Top- Coder deem the platform has played at least a moderate role in improving their programming ability. What s more, TopCoder would announce Coder of the Month and top ten developers of each types of tasks on their website to satisfy developers desire for recognition. These non-monetary incentives can be even more important than the money reward when motivating crowd developers for participation. B. Recommendation Systems for Software Development Recommendation systems for software development support developers who need to make decisions and choose from among a potentially overwhelmingly large set of possibilities. Examples of recommendation systems supporting software development include code navigation support, software change guidance, developer-bug assignment. An overview of available recommendation systems for software engineering can be found in the work of Robillard et al. [15]. Fig. 3. Crowdsourced software development process and its task phases (derived from TopCoder.com and Mao et al. [10]). 3 The scorecards are available at: 4 The online survey can be accessed at: 349
4 Recommendation systems often employ either contentbased or collaborative techniques (or use hybrids) [16]. In this paper we focus on content-based methods for developer recommendation. The fundamental assumption of content-base recommendation methods is that the user will be interested in the items that are similar (in some measurable sense) to the items the user preferred in the past. In the context of developer recommendation, in order to recommend developers for a newly arriving task, the content-based recommender tries to understand the commonalities among the tasks that the developers had participated. Then those developers whose commonalities are best matched with the newly arriving task would be recommended. III. THE APPROACH TO DEVELOPER RECOMMENDATION FOR CSD TASKS Our recommender system recommends developers in order to satisfy two objectives. One target is to recommend reliable developers for delivering qualified software assets. The other target is to recommend suitable developers who may be interested in registration for promoting the task participation level, and meanwhile reducing their effort in task selection. A. Our Proposed Developer Recommendation Framework The framework of our approach is named CrowdRex, which is shown in Figure 4. Our approach is based on content: a group of features describing a CSD task, such as required techniques, payment, title and overview description, post data and submission deadline, etc. The machine learner in the framework tries to capture the characteristics (e.g., expertise, expected payment and task duration, etc.) of each developer from their historical activities and recommend the best matched developers to newly arriving tasks. In order to satisfy Fig. 4. CrowdRex: our developer recommendation framework. the two targets described above, our approach is designed to extract empirical knowledge from developers winning and participation history separately to suggest two lists of developers corresponding to the objectives. The final list of recommended developers can adopt the two lists directly or use a re-rank method (e.g., prioritise developers according to their recent performance) to select a subset of developers according to the application context. In this paper we adopt the two lists of developers as our recommendation results. B. Recommend Developers for Delivering Qualified Assets In order to help the platform find reliable developers, we recommend skilled developers to undertake their familiar tasks. Our assumption here is that skilled developers undertaking familiar tasks are more likely to deliver qualified software assets. To identify a qualified software asset, TopCoder has a review system (introduced in Section II) to guarantee software quality, which requires a qualified asset to pass a minimum review score. Generally, for the historical tasks, if there exists a winner, that means a qualified asset can be delivered. In this paper, we define a developer as skilled if he or she has delivered at least 5 qualified assets. We employ multi-class, single-label classification technique to automate the process of identifying developers who may be familiar with the newly arriving tasks. Here we treat developer as class. Each task in historical records is labelled with the first place winner (the second place winner is not considered as the label is missing in many records). The recommendation approach consists of four major phases: 1) Data Filtering: We filter those historical tasks (i.e. training set) with incomplete information (e.g., missing winner label or lack of task descriptions). To exclude irrelevance empirical knowledge, we remove the tasks which development type do not match current application domain (e.g., development, assembly, etc.). Also, since we treat a developer who won at least 5 tasks as a skill developer, those historical records with labelled winners who fail to satisfy this criteria are filtered out. 2) Feature Extraction: CSD tasks on TopCoder contain a variety of information. For feature extraction in this paper we consider the informative text features including title, description, programming language and techniques. Numeric features include task post date, allocated task duration and payment. The detailed feature description is presented in Table I. In order to unify numeric and text features, we convert each text feature into word vector format, keeping only meaningful and descriptive tokens processed by tokenization and stop words (e.g., a, the, and, of, is, this, etc.) removal. More formally, suppose there are m terms after the process for a text feature, the corresponding vector of this feature in task t T would be v t =(w t,1,w t,2,..., w t,m ), where w i,j stands for the weight for each term term j, which is calculated by Term Frequency-Inverse Document Frequency (TF-IDF) according 350
5 TABLE I THE CONTENT FEATURES FOR CROWDSOURCED SOFTWARE TASKS Feature Format Description Date Numeric Post date of the task. PL Text What Programming Language is used. Title Text Title of the posted task. Tech Text Indicate what techniques are used. Description Text Task descriptions overview. Duration Numeric Time allocated to the task. Payment Numeric How much US dollars will the winner get. to Equation 1: w i,j = ( 1+log(tf j ) ) log T (1) df i Where tf j is the number of times term j appears in task i, and T is the total number of tasks and df i equals the number of tasks containing term j. 3) Learner Training: In this step we train a classification model on prepared historical tasks. Various supervised learning algorithms can be adopted in this step, to generate a distribution of probabilities for all labels. Such algorithms can be C4.5 Decision Tree [17], NaïveBayes [18], k-nearest Neighbour [19], etc. 4) Learner Applying: Lastly, we apply the trained model on test set, i.e. newly arriving tasks. Top N ranked winners from the generated probability distribution are recommended. Note that the k-nearest Neighbour algorithm mentioned in phase 3) is slightly different from other supervised learners because it does not have an explicit model training phase. However the lazy nature of this method makes it easier to understand and previous research have shown its efficiency and effectiveness. In order to use the k-nearest Neighbour algorithm, we need to define the similarity measure. In this paper we define the similarity Sim between two task t i and t j according to the post date distance, matching in task programming language, matched number of techniques, allocated duration distance, payment difference and the text matching degree (i.e. cosine similarity of two vectors) in title and description. Sim is defined as Equation 2. Sim(t i,t j )=w 1 Dis(F 1,i,F 1,j )+w 2 Dis(F 2,i,F 2,j ) (2) +w 3 Dis(F 3,i,F 3,j ) w n Dis(F n,i,f n,j ) Where w stands for the weight assigned to the corresponding feature (in this paper we use equal weights i.e. 1.0 for all features), Dis indicates the distance function which varies among different features. The definition of distance measures in this paper is shown in Table II. C. Recommend Developers for Participation By recommending developers for participation, we aim to enhance task participation level by helping developers to receive information on newly arriving tasks which are suitable for them, meanwhile this can save their effort in task selection. Admittedly, compare with those reliable developers in the winner group, these long-tail participants have much less chance to win the task prize. However their demand for participation still exists thus the participant recommendation service can be useful. We define registrants of each historical task as its participants in this paper. As there are multiple participants who register the newly arriving task to show their willingness in engaging with the work. We treat developer recommendation for participation as a multi-class, multi-label classification problem. Previous research have suggested lots of methods to solve this problem. Tsoumakas and Katakis [20] grouped existing methods into two categories: problem transformation methods and algorithm adaptation methods. In this paper we use a straightforward data transform based method which fall into the first category. The method works as follows (the data filtering and feature extraction processes are the same as described above, which are omitted here): Firstly, for each training historical task record with participants set P, decomposes the record into P records, each of these records keeps the task features but corresponds to only one distinct participant p P. Then the problem is transformed to a multi-class, single-label classification problem. In order to recommend multiple participants, we apply a single-label machine learner (e.g., NaïveBayes) that can generate a distribution of probabilities for all labels. Finally use the trained model to process the testing records, rank the participant labels according to the generated probability distribution and recommend top N developers. IV. EMPIRICAL STUDY This section presents the empirical study conducted on two datasets with different development characteristics to evaluate our proposed developer recommendation approach for CSD tasks. Since there is no existing study that evaluated any approach for CSD recommendation, we employ a simple but actionable method named Active as our baseline, which Feature Date TABLE II THE ADOPTED FEATURE DISTANCE MEASURES Distance Measure (Date i Date j )/Date MaxDiff PL PL i == PL j?1:0 Title Tech Description Duration Payment Tit x Tit y Tit x Tit y Match(Tech i,tech j )/N umberof T echs Max Des x Des y Des x Des y (Duration i Duration j )/Duration Max (Payment i Payment j )/P ayment Max 351
6 TABLE III STATISTICS OF THE EVALUATION DATASETS Dataset # Tasks # Reg. # Win. Duration Development 1093/ / / Assembly 1505/ / / Task Count Task Count Assembly Number of Registrants Development Fig. 5. The distribution on the number of registrants in each task. recommends the top N developers who have won most tasks for delivering qualified assets, and recommends the top N developers who have registered most tasks for participation, according to the given training dataset. Through the evaluation, we empirically study the following three research questions: RQ1. (Baseline Comparison): How does our proposed recommendation approach (instantiated with different machine learners) perform, when compared to the Active method and assessed from accuracy and diversity? Since the baseline approach is simple and actionable, if we cannot outperform this method then there is no reason for the platform to adopt our recommender system. RQ2. (Performance Assessment): Which of the machine learners can best support our recommender system according to accuracy and diversity metrics? Can the best learner consistently outperform other learners, on different datasets and for different recommendation purposes? RQ3. (Insights): What useful insights from the performance data can we yield for the real world CSD platform? A. Dataset We evaluate our approach on the novel datasets collected from TopCoder, which currently has the largest community for crowdsourced software development. We collected 3,094 historical tasks that have been crowdsourced between Oct and Mar. 2013, covering 2 types of software development tasks (component development and assembly). We evaluate our approach separately on each of the 2 datasets. Summary data regarding the 2 datasets after and before data filtering is shown in Table III. Note that the 3rd and 4th columns indicate the number of distinct registrants and winners respectively. Figure 5 presents the registrations distribution among the 2 types of CSD tasks. The statistics generally indicates a normal distribution for the number of registrants in each task. B. Study Setting We sort the task records ascendingly according to their posted time and then divide each of the 2 datasets into 10 folds. The top 9 folds of records are selected as our training set and the 10th fold is selected as our testing set, i.e. newly arriving tasks. On each of these datasets, we recommend top 5, 10 and 20 developers separately and record the recommended lists for calculating evaluation metrics. Note that here we choose to recommend 5, 10 and 20 developers, which produce larger sets of candidate developers compare to most studies on bug triage. This is because in the CSD context, a wider range of developers are encouraged to participate, and the parameters of top 5, 10 and 20 developers basically matches the real registrants distribution shown in Figure 5: For component development tasks, the average registrants number is 16, and the 10th-90th percentile corresponds to registrants. For assembly tasks, there are 20 registrants on average for a task and the 10th-90th percentile corresponds to 9-36 registrants. Besides, Figure 6 presents the relationship between the number of registrants and number of qualified submissions. We can see that at least 5 registrants are expected in order to receive one submission for a task. Also, 10 and 20 registrants are expected to obtain 2 submissions for the development and assembly tasks separately. We choose to recommend 5, 10 and 20 developers for simulating the real application of our approach. The machine learning algorithms evaluated in our study are selected from popular algorithms applied in previous related research, which include C4.5 Decision Tree (C4.5), NaïveBayes (NB) and k-nearest Neighbour when k = 1 (KNN 1) and k =5(KNN 5). For these machine learners applied in the experiments, we do not optimise their parameters for a certain method to avoid biasing. Our results are therefore directly compared to the baseline, which also requires no tuning. The corresponding parameters are set according to the default parameters in the popular open sourced data mining tool named Weka [21]. C. Evaluation Metrics In order to evaluate the performance of our proposed approach, we need metrics that can be used to assess developer recommendation techniques. Previous related research topics Number of submissions Development Assembly Number of registrants Fig. 6. The relationship between the number of registrants and submissions. 352
7 TABLE IV ACCURACY AND DIVERSITY OF DEVELOPER RECOMMENDATION FOR DELIVERING QUALIFIED SOFTWARE ASSETS Dataset DEV ASM Recommend C4.5 NaïveBayes KNN 1 KNN 5 Active Acc Div Acc Div Acc Div Acc Div Acc Div 5 50% 40% 44% 21% 34% 27% 32% 35% 37% 5% 10 60% 45% 54% 26% 44% 30% 49% 38% 46% 11% 20 71% 52% 67% 18% 56% 40% 61% 42% 57% 22% 5 37% 72% 33% 33% 38% 47% 43% 73% 15% 6% 10 51% 74% 44% 18% 49% 49% 58% 76% 26% 12% 20 63% 77% 54% 48% 57% 53% 67% 78% 37% 23% TABLE V ACCURACY AND DIVERSITY OF DEVELOPER RECOMMENDATION FOR PARTICIPATION Dataset DEV ASM Recommend C4.5 NaïveBayes KNN 1 KNN 5 Active Acc Div Acc Div Acc Div Acc Div Acc Div 5 30% 3% 4% 15% 3% 8% 3% 7% 24% 1% 10 23% 6% 2% 17% 5% 14% 6% 13% 18% 1% 20 21% 11% 1% 18% 6% 20% 6% 19% 12% 2% 5 69% 1% 12% 10% 10% 35% 10% 34% 65% 1% 10 48% 2% 6% 11% 16% 47% 17% 46% 44% 2% 20 30% 4% 3% 12% 17% 53% 18% 53% 25% 4% such as bug triage focus on the Accuracy dimension of the performance, the typical metrics used are Precision and Recall. However in the context of CSD tasks, we additionally need to consider Diversity, which aims to encourage more developers to migrate into the task as the essence of crowdsourcing is to utilise the wisdom of the crowd. A recommender system can be statistically accurate if we focus on optimising the Accuracy dimension but it may be not very useful for practical purpose if it has poor diversity. For example, suppose a developer d participates in all the CSD tasks and the developer recommender would recommend d to any newly arriving tasks in order to be accurate. However since d is so active, he or she would likely still participate the newly arriving tasks without them being recommended. If the recommender system focus only on accuracy, it will fail to enhance any participation level. For measuring the Diversity dimension, we follow the metric used by Adomavicius and Kwon [22]. It is calculated based on the number of distinct developers that can ever be recommended by the recommender system. To normalise the diversity measure, we define it as the percentage format. This diversity metric can be calculated according to Equation 3 where R i (t)) stands for the recommended i developers for task t, Actual (t) stands for the ground truth developers for task t, Tr stands for training set and T stands for testing set. As for the Accuracy metrics, we define HitRate i to assess the recommender for delivering qualified assets (one groundtruth-label/winner for each task) and use AverageP recision to assess the recommender for participation (multiple groundtruth-labels/registrants for each task), i.e. the Accuracy dimension in Table IV is measured from HitRate and the Accuracy dimension in Table V is measured from AverageP recision. These two metrics are defined formally as shown in Equation 4 and Equation 5. / Div i = R i (t) Actual (t) t T r t T (3) HitRate i = 1 T correct ( R i (t) ) (4) t T AvgP rec i = 1 T ( correct ( R i (t) ) / R i (t) ) (5) t T D. Results and Analysis Table IV shows the results for delivering qualified assets evaluated on the datasets of Component Development (DEV) and Assembly (ASM) CSD tasks. Table V presents the results for participant recommendation on these two datasets. A performance comparison of the results in above tables is illustrated by the charts in Figure 7. 1) Results for RQ1 (Baseline Comparison): The performance of our recommender system outperforms the baseline Active method across all datasets with different settings (recommending 5, 10, 20 developers). When recommending developers for delivering qualified assets, our recommender 353
8 Fig. 7. Performance comparison of developer recommendation when recommending 5, 10 and 20 developers. The x-axis shows the machine learners and the baseline method Active. The y-axis shows the value for Accuracy (Acc) and Diversity (Div) measures. The scatter points are linked for the purpose of improving the readability. system instantiated with any of the 4 machine learners performs better than the baseline method for most cases, assessing from Accuracy and Diversity. When recommending developers for participation, since the baseline method shows good capability to recommend developers accurately, the best performed learner C4.5, on average is only 5.5% better than baseline from the Accuracy perspective, and from the Diversity perspective, our recommender employ different machine learner shows different capabilities, i.e. the KNN 1 (8%-53%), KNN 5 (7%- 53%) and NaïveBayes (10%-18%) learners performs much better than the baseline (1%-4%) method. 2) Results for RQ2 (Performance Assessment): As shown in Table IV and Table V, there does not exist a machine learner that can consistently support our recommender system better than other machine learners. However the C4.5 decision tree learner performs best for Accuracy in 9 of all 12 cases which indicates that it is the best learner for Accuracy. Evaluating from the Diversity dimension, KNN 1 is the best learner which outperforms others in 4 of 12 cases. When we assess from both Accuracy and Diversity dimensions by calculating their Harmonic mean (i.e. 2 Accuracy Diversity/(Accuracy + Diversity)), C4.5 decision tree outperforms others in 5 of 12 cases. Thus we regard C4.5 as the best learner for our proposed recommender system, when assessing from the Harmonic mean of Accuracy and Diversity measures. From Figure 7 we can see that when recommending developers for delivering qualified assets, our proposed recommender system can achieve reasonable accuracy and diversity. However we cannot reach both high accuracy and diversity when recommending developers for participation. This reveals an Accuracy-Diversity dilemma of our recommender system. Observed from the performance of the baseline Active method, when recommending developers for participation, the biased learner with high accuracy and low diversity can be attributed to the ground truth of the participants distribution, i.e., there does exist the phenomenon that a certain number of active developers tend to participate newly arriving tasks persistently. Admittedly, the recommender system with high accuracy and low diversity consistently recommend those active developers can be a solution, but this only works for short-term benefit. Only by achieving reasonable diversity, can the recommender serve for the long-tail developers and enhance the task participation level (i.e., exploration). Meanwhile, the recommendation can recommend some of the active developers to ensure the accuracy is acceptable (i.e., exploitation). We suggest that a proper trade-off between exploration and exploitation may help the recommender system work for long-term benefit. 3) Results for RQ3 (Insight): Our empirical results can be used to derive a few insights which are listed as follows: 1). In conclusion, from the performance data in Table IV and Table V, we suggest some careful selection from among different machine learners will be required according the application context (e.g., task type). This conclusion is consistent with the no-free-lunch theorem [23] for machine learning. 2). Those most active developers are very likely to participate in newly arriving tasks. However they are only a small proportion of all developers, shown from the high accuracy and low diversity of Active method. This indicates that, to enhance the participation level, the CSD platform may need to allocate higher weight to diversity to focus on recommending suitable, but not necessarily the most active developers. The winners of newly arriving tasks tend to be more various according to the high diversity in Table V, which may indicate that skilled developers have different knowledge background and tend to deliver qualified assets for those matched tasks within their expertise. 3). The baseline Active method is simple and actionable, but can be an option to recommend developers for participation 354
9 if the low diversity can be tolerated. It may yield short-term advantage at the expose of the long-term sustainability of the CSD platform. V. RELATED WORK Crowdsourced Software Development (CSD) has received increasing attention from the research community in recent years. The research topics vary from CSD model to domain application. Kazman and Chen [24] argued that traditional software development life-cycle models such as the waterfall model and the spiral model are inadequate for mass peer production and the service nature of crowdsourcing. They proposed the Metropolis model for the development of crowdsourced systems. For the practice of CSD, Prikladnicki et al. [25] reported their starting point of a multi-year study on CSD in Brazil, which aims to identify the challenges as well as the best practices for CSD. Musson et al. [26] showed how they utilised the crowd users to improve the Lync software performance at Microsoft. Crowdsouring can also be used to tackle classical problems in software engineering. Schiller and Ernst [27] presented VeriWeb, which is an online IDE for solving the skill barriers problem in verification. Their experiments showed that VeriWeb can lower the monetary and time cost of verification. However the workers need to be professional contract workers rather than ad-hoc workers. Pastore et al. [28] conducted a study on using the crowd to solve the oracle problem [29] in software testing. Their experimental results indicated that although it still remains hard to obtain qualified solutions from the crowd, it can be a viable method to ameliorate the oracle problem. For the quality concern of CSD outcomes, Li et al. [9] analysed the key factors on CSD software quality. By conducting an empirical study on TopCoder.com, they suggested four critical factors in improving the quality of the deliverables, including the prosperous period (related to the number of active developers, developers ratings, etc.) of the CSD platform, the size of the task, the maximum skill level of the crowd developers as well as the software design quality of the CSD task. Our review of the literature reveals that few previous authors have investigated recommendation-related topics for the emerging crowdsourcing paradigms. We could not find any peer-reviewed papers on developer recommendation in the CSD context. However, for the more general context of recommending crowdsourced micro-tasks to users, we did find related work although this is not directly related to CSD recommender systems. Ambati, Vogel and Carbonell [30] proposed a recommendation approach based on implicit modelling of interest and skills. Yuen, King and Leung [31] employed probabilistic matrix factorization for preferencebased task recommendation in crowdsourcing system. These two studies were conducted on the Amazon Mechanical Turk platform with micro-tasks, rather than tasks as complex as software development. Another related topic which has been studied more extensively is bug triage. Čubranić and Murphy [32] proposed to automate bug triage using text categorization techniques. Anvik, Hiew and Murphy [33] expanded the previous work by data pre-processing, using additional features and exploring the performance of more machine learning algorithms. Gaeul, Kim and Thomas [34] further improved bug triage accuracy using developers tossing behaviours in bug repositories. Matter, Kuhn and Niestrasz [35] presented their approach to automatically suggest developers for handling bug reports using a vocabulary-based expertise model. Tamrawi et al. [36] introduced fuzzy set and cache-based modelling techniques for automatic bug triage. Xuan et al. [37] leveraged developer prioritization based on a social network techniques to assist triage tasks in bug repositories. Wang et al. [38] explored developers collaboration relationship in bug repositories. They model the collaboration as heterogeneous graphs to help triage the bug reports and their evaluation showed higher accuracies can be achieved by utilising the heterogeneous graphs. However these proposed approaches need to be carefully re-examined in the CSD context, due to the specialized properties and peculiarities of CSD which require special treatment. For instance, one prior research has shown that traditional laws on software cost are challenged in the CSD context [10]. Likewise, we consider traditional bug triage methods may not migrate to this new context directly because previous bug triage studies are based on open source communities, which are quite different from crowdsourcing community. VI. SUMMARY Crowdsourced software development is an emerging paradigm which utilises the wisdom of the crowd for software production. The CSD tasks demand reliable developers and enough participation to guarantee a qualified asset that can be delivered to the client. In addition, examining from hundreds of tasks descriptions, it is quite a laborious and demanding work for the developers to choose which one is more suitable for them to participate. To tackle above challenges, we introduced content-based recommendation techniques for developer recommendation in the emerging CSD context. Our system learns particular interests from registration history and mines winner history to favour appropriate developers. To encourage task participation as well as to serve more developers, we focus not only accuracy but also diversity to measure the performance of our recommender system. The experimental results show that our recommender system outperforms the baseline method and can achieve promising accuracy and diversity. We also provide a few insights concluded from the results for the CSD platform and its online developers. For future work we plan to propose an improved approach to optimise recommendation results by considering developers strategic behaviours and their social network information. A wider range of machine learners and related state-of-the-art methods in bug triage would be evaluated for performance comparison. In addition, to further investigate the usefulness of the recommender system, an online empirical study would be conducted. 355
10 ACKNOWLEDGMENT This research is funded by UCL Graduate Research Scholarships (GRS), UCL Overseas Research Scholarships (ORS) and Dynamic Adaptive Automated Software Engineering (DAASE) programme grant (EP/J017515). The authors would like to thank TopCoder for giving us the permission to use their CSD task data for this work. REFERENCES [1] A. Begel, J. D. Herbsleb, and M.-A. D. Storey, The future of collaborative software development. in CSCW (Companion), S. E. Poltrock, C. Simone, J. Grudin, G. Mark, and J. Riedl, Eds. ACM, 2012, pp [2] J. Howe, The rise of crowdsourcing, Wired magazine, vol. 14, no. 6, pp. 1 4, [3] Massolution, Crowdsourcing industry report, [4] N. Archak, Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on topcoder.com. in WWW, M. Rappa, P. Jones, J. Freire, and S. Chakrabarti, Eds. ACM, 2010, pp [5] K. Lakhani, D. Garvin, and E. Lonstein, Topcoder (a): Developing software through crowdsourcing, Harvard Business School Case, no , [6] A. Begel, J. Bosch, and M.-A. D. Storey, Social networking meets software development: Perspectives from github, msdn, stack exchange, and topcoder. IEEE Software, vol. 30, no. 1, pp , [7] L. B. Chilton, J. J. Horton, R. C. Miller, and S. Azenkot, Task search in a human computation market, in Proceedings of the ACM SIGKDD Workshop on Human Computation, ser. HCOMP 10. New York, NY, USA: ACM, 2010, pp [8] V. Wadhwa and A. R. Ravindran, Vendor selection in outsourcing, Computers & Operations Research, vol. 34, no. 12, pp , [9] K. Li, J. Xiao, Y. Wang, and Q. Wang, Analysis of the key factors for software quality in crowdsourcing development: An empirical study on topcoder.com. in COMPSAC. IEEE Computer Society, [10] K. Mao, Y. Yang, M. Li, and M. Harman, Pricing crowdsourcing-based software development tasks, in Proceedings of the 2013 International Conference on Software Engineering, New Ideas and Emerging Results Track, ser. ICSE IEEE Press, 2013, pp [11] J. Antin and A. Shaw, Social desirability bias and self-reports of motivation: A study of amazon mechanical turk in the us and india, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI 12. New York, NY, USA: ACM, 2012, pp [12] N. Kaufmann, T. Schulze, and D. Veit, More than fun and money. worker motivation in crowdsourcing: a study on mechanical turk, in Proceedings of the Seventeenth Americas Conference on Information Systems, 2011, pp [13] M.-C. Yuen, I. King, and K.-S. Leung, A survey of crowdsourcing systems, in Privacy, Security, Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on, Oct 2011, pp [14] Jim McKeown, Open source meets capitalism: Collaborate by competing. [Online]. Available: [15] M. P. Robillard, R. J. Walker, and T. Zimmermann, Recommendation systems for software engineering, IEEE Software, vol. 27, no. 4, pp , July/August [16] G. Adomavicius and A. Tuzhilin, Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions, IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 6, pp , June [17] J. R. Quinlan, C4.5: programs for machine learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., [18] P. Langley, W. Iba, and K. Thompson, An analysis of bayesian classifiers, in In Proceedings of the 10th International Conference on Artificial Intelligence. MIT Press, 1992, pp [19] T. Cover and P. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory, vol. 13, no. 1, pp , [20] G. Tsoumakas and I. Katakis, Multi-label classification: An overview, Int J Data Warehousing and Mining, vol. 2007, pp. 1 13, [21] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, The weka data mining software: an update, SIGKDD Explor. Newsl., vol. 11, no. 1, pp , Nov [22] G. Adomavicius and Y. Kwon, Toward more diverse recommendations: Item re-ranking methods for recommender systems, in Proceedings of the 19th Workshop on Information Technology and Systems (WITS 09), Dec [23] D. H. Wolpert and W. G. Macready, No free lunch theorems for optimization, Evolutionary Computation, IEEE Transactions on, vol. 1, no. 1, pp , [24] R. Kazman and H.-M. Chen, The metropolis model a new logic for development of crowdsourced systems, Commun. ACM, vol. 52, no. 7, pp , Jul [25] R. Prikladnicki, L. Machado, E. Carmel, and C. R. B. de Souza, Brazil software crowdsourcing: A first step in a multi-year study, in Proceedings of the 1st International Workshop on CrowdSourcing in Software Engineering, ser. CSI-SE New York, NY, USA: ACM, 2014, pp [26] R. Musson, J. Richards, D. Fisher, C. Bird, B. Bussone, and S. Ganguly, Leveraging the crowd: How 48,000 users helped improve lync performance, IEEE Softw., vol. 30, no. 4, pp , Jul [27] T. W. Schiller and M. D. Ernst, Reducing the barriers to writing verified specifications, in Object-Oriented Programming Systems, Languages, and Applications (OOPSLA 2012), Tucson, AZ, USA, October 23-25, 2012, pp [28] F. Pastore, L. Mariani, and G. Fraser, Crowdoracles: Can the crowd solve the oracle problem? in Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, ser. ICST 13. Washington, DC, USA: IEEE Computer Society, 2013, pp [29] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, The oracle problem in software testing: A survey, IEEE Transactions on Software Engineering, 2015, to appear. [30] V. Ambati, S. Vogel, and J. G. Carbonell, Towards task recommendation in micro-task markets. in Human Computation, ser. AAAI Workshops, vol. WS AAAI, [31] M.-C. Yuen, I. King, and K.-S. Leung, Taskrec: probabilistic matrix factorization in task recommendation in crowdsourcing systems, in Neural Information Processing. Springer, 2012, pp [32] D. ubrani, Automatic bug triage using text categorization, in In SEKE 2004: Proceedings of the Sixteenth International Conference on Software Engineering & Knowledge Engineering. KSI Press, 2004, pp [33] J. Anvik, L. Hiew, and G. C. Murphy, Who should fix this bug? in Proceedings of the 28th International Conference on Software Engineering. ACM, 2006, pp [34] G. Jeong, S. Kim, and T. Zimmermann, Improving bug triage with bug tossing graphs, in Proceedings of the the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT symposium on The Foundations of Software Engineering, ser. ESEC/FSE 09. New York, NY, USA: ACM, 2009, pp [35] D. Matter, A. Kuhn, and O. Nierstrasz, Assigning bug reports using a vocabulary-based expertise model of developers, in Proceedings of the th IEEE International Working Conference on Mining Software Repositories, ser. MSR 09. Washington, DC, USA: IEEE Computer Society, 2009, pp [36] A. Tamrawi, T. T. Nguyen, J. M. Al-Kofahi, and T. N. Nguyen, Fuzzy set and cache-based approach for bug triaging, in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European Conference on Foundations of Software Engineering, ser. ESEC/FSE 11. New York, NY, USA: ACM, 2011, pp [37] J. Xuan, H. Jiang, Z. Ren, and W. Zou, Developer prioritization in bug repositories, in Proceedings of the 2012 International Conference on Software Engineering, ser. ICSE Piscataway, NJ, USA: IEEE Press, 2012, pp [38] S. Wang, W. Zhang, Y. Yang, and Q. Wang, Devnet: Exploring developer collaboration in heterogeneous networks of bug repositories, in Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on, Oct 2013, pp
Bug triage in open source systems: a review
Int. J. Collaborative Enterprise, Vol. 4, No. 4, 2014 299 Bug triage in open source systems: a review V. Akila* and G. Zayaraz Department of Computer Science and Engineering, Pondicherry Engineering College,
More informationReducing Features to Improve Bug Prediction
Reducing Features to Improve Bug Prediction Shivkumar Shivaji, E. James Whitehead, Jr., Ram Akella University of California Santa Cruz {shiv,ejw,ram}@soe.ucsc.edu Sunghun Kim Hong Kong University of Science
More informationRule Learning With Negation: Issues Regarding Effectiveness
Rule Learning With Negation: Issues Regarding Effectiveness S. Chua, F. Coenen, G. Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX Liverpool, United
More informationA Case Study: News Classification Based on Term Frequency
A Case Study: News Classification Based on Term Frequency Petr Kroha Faculty of Computer Science University of Technology 09107 Chemnitz Germany kroha@informatik.tu-chemnitz.de Ricardo Baeza-Yates Center
More informationLearning From the Past with Experiment Databases
Learning From the Past with Experiment Databases Joaquin Vanschoren 1, Bernhard Pfahringer 2, and Geoff Holmes 2 1 Computer Science Dept., K.U.Leuven, Leuven, Belgium 2 Computer Science Dept., University
More informationEvaluation of Usage Patterns for Web-based Educational Systems using Web Mining
Evaluation of Usage Patterns for Web-based Educational Systems using Web Mining Dave Donnellan, School of Computer Applications Dublin City University Dublin 9 Ireland daviddonnellan@eircom.net Claus Pahl
More informationEvaluation of Usage Patterns for Web-based Educational Systems using Web Mining
Evaluation of Usage Patterns for Web-based Educational Systems using Web Mining Dave Donnellan, School of Computer Applications Dublin City University Dublin 9 Ireland daviddonnellan@eircom.net Claus Pahl
More informationRule Learning with Negation: Issues Regarding Effectiveness
Rule Learning with Negation: Issues Regarding Effectiveness Stephanie Chua, Frans Coenen, and Grant Malcolm University of Liverpool Department of Computer Science, Ashton Building, Ashton Street, L69 3BX
More informationModule 12. Machine Learning. Version 2 CSE IIT, Kharagpur
Module 12 Machine Learning 12.1 Instructional Objective The students should understand the concept of learning systems Students should learn about different aspects of a learning system Students should
More informationThe Good Judgment Project: A large scale test of different methods of combining expert predictions
The Good Judgment Project: A large scale test of different methods of combining expert predictions Lyle Ungar, Barb Mellors, Jon Baron, Phil Tetlock, Jaime Ramos, Sam Swift The University of Pennsylvania
More informationCoordination Challenges in Global Software Development
Coordination Challenges in Global Software Development Anusuyah Subbarao, Dr Mohd Naz ri Mahrin Advanced Informatics School, Universiti Teknologi Malaysia, Jalan Sultan Yahya Petra, 54100 Kuala Lumpur,
More informationProbabilistic Latent Semantic Analysis
Probabilistic Latent Semantic Analysis Thomas Hofmann Presentation by Ioannis Pavlopoulos & Andreas Damianou for the course of Data Mining & Exploration 1 Outline Latent Semantic Analysis o Need o Overview
More informationAssignment 1: Predicting Amazon Review Ratings
Assignment 1: Predicting Amazon Review Ratings 1 Dataset Analysis Richard Park r2park@acsmail.ucsd.edu February 23, 2015 The dataset selected for this assignment comes from the set of Amazon reviews for
More informationSystem Implementation for SemEval-2017 Task 4 Subtask A Based on Interpolated Deep Neural Networks
System Implementation for SemEval-2017 Task 4 Subtask A Based on Interpolated Deep Neural Networks 1 Tzu-Hsuan Yang, 2 Tzu-Hsuan Tseng, and 3 Chia-Ping Chen Department of Computer Science and Engineering
More informationSINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF)
SINGLE DOCUMENT AUTOMATIC TEXT SUMMARIZATION USING TERM FREQUENCY-INVERSE DOCUMENT FREQUENCY (TF-IDF) Hans Christian 1 ; Mikhael Pramodana Agus 2 ; Derwin Suhartono 3 1,2,3 Computer Science Department,
More informationVisit us at:
White Paper Integrating Six Sigma and Software Testing Process for Removal of Wastage & Optimizing Resource Utilization 24 October 2013 With resources working for extended hours and in a pressurized environment,
More informationLecture 1: Machine Learning Basics
1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3
More informationOn the Combined Behavior of Autonomous Resource Management Agents
On the Combined Behavior of Autonomous Resource Management Agents Siri Fagernes 1 and Alva L. Couch 2 1 Faculty of Engineering Oslo University College Oslo, Norway siri.fagernes@iu.hio.no 2 Computer Science
More informationNotes on The Sciences of the Artificial Adapted from a shorter document written for course (Deciding What to Design) 1
Notes on The Sciences of the Artificial Adapted from a shorter document written for course 17-652 (Deciding What to Design) 1 Ali Almossawi December 29, 2005 1 Introduction The Sciences of the Artificial
More informationA cognitive perspective on pair programming
Association for Information Systems AIS Electronic Library (AISeL) AMCIS 2006 Proceedings Americas Conference on Information Systems (AMCIS) December 2006 A cognitive perspective on pair programming Radhika
More informationRule discovery in Web-based educational systems using Grammar-Based Genetic Programming
Data Mining VI 205 Rule discovery in Web-based educational systems using Grammar-Based Genetic Programming C. Romero, S. Ventura, C. Hervás & P. González Universidad de Córdoba, Campus Universitario de
More informationPredicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks
Predicting Student Attrition in MOOCs using Sentiment Analysis and Neural Networks Devendra Singh Chaplot, Eunhee Rhim, and Jihie Kim Samsung Electronics Co., Ltd. Seoul, South Korea {dev.chaplot,eunhee.rhim,jihie.kim}@samsung.com
More informationSpeech Emotion Recognition Using Support Vector Machine
Speech Emotion Recognition Using Support Vector Machine Yixiong Pan, Peipei Shen and Liping Shen Department of Computer Technology Shanghai JiaoTong University, Shanghai, China panyixiong@sjtu.edu.cn,
More informationLearning Methods for Fuzzy Systems
Learning Methods for Fuzzy Systems Rudolf Kruse and Andreas Nürnberger Department of Computer Science, University of Magdeburg Universitätsplatz, D-396 Magdeburg, Germany Phone : +49.39.67.876, Fax : +49.39.67.8
More informationTerm Weighting based on Document Revision History
Term Weighting based on Document Revision History Sérgio Nunes, Cristina Ribeiro, and Gabriel David INESC Porto, DEI, Faculdade de Engenharia, Universidade do Porto. Rua Dr. Roberto Frias, s/n. 4200-465
More informationTruth Inference in Crowdsourcing: Is the Problem Solved?
Truth Inference in Crowdsourcing: Is the Problem Solved? Yudian Zheng, Guoliang Li #, Yuanbing Li #, Caihua Shan, Reynold Cheng # Department of Computer Science, Tsinghua University Department of Computer
More informationRadius STEM Readiness TM
Curriculum Guide Radius STEM Readiness TM While today s teens are surrounded by technology, we face a stark and imminent shortage of graduates pursuing careers in Science, Technology, Engineering, and
More informationA GENERIC SPLIT PROCESS MODEL FOR ASSET MANAGEMENT DECISION-MAKING
A GENERIC SPLIT PROCESS MODEL FOR ASSET MANAGEMENT DECISION-MAKING Yong Sun, a * Colin Fidge b and Lin Ma a a CRC for Integrated Engineering Asset Management, School of Engineering Systems, Queensland
More informationThree Strategies for Open Source Deployment: Substitution, Innovation, and Knowledge Reuse
Three Strategies for Open Source Deployment: Substitution, Innovation, and Knowledge Reuse Jonathan P. Allen 1 1 University of San Francisco, 2130 Fulton St., CA 94117, USA, jpallen@usfca.edu Abstract.
More informationA Pipelined Approach for Iterative Software Process Model
A Pipelined Approach for Iterative Software Process Model Ms.Prasanthi E R, Ms.Aparna Rathi, Ms.Vardhani J P, Mr.Vivek Krishna Electronics and Radar Development Establishment C V Raman Nagar, Bangalore-560093,
More informationSoftware Maintenance
1 What is Software Maintenance? Software Maintenance is a very broad activity that includes error corrections, enhancements of capabilities, deletion of obsolete capabilities, and optimization. 2 Categories
More informationThe 9 th International Scientific Conference elearning and software for Education Bucharest, April 25-26, / X
The 9 th International Scientific Conference elearning and software for Education Bucharest, April 25-26, 2013 10.12753/2066-026X-13-154 DATA MINING SOLUTIONS FOR DETERMINING STUDENT'S PROFILE Adela BÂRA,
More informationTowards a Collaboration Framework for Selection of ICT Tools
Towards a Collaboration Framework for Selection of ICT Tools Deepak Sahni, Jan Van den Bergh, and Karin Coninx Hasselt University - transnationale Universiteit Limburg Expertise Centre for Digital Media
More informationWE GAVE A LAWYER BASIC MATH SKILLS, AND YOU WON T BELIEVE WHAT HAPPENED NEXT
WE GAVE A LAWYER BASIC MATH SKILLS, AND YOU WON T BELIEVE WHAT HAPPENED NEXT PRACTICAL APPLICATIONS OF RANDOM SAMPLING IN ediscovery By Matthew Verga, J.D. INTRODUCTION Anyone who spends ample time working
More informationWord Segmentation of Off-line Handwritten Documents
Word Segmentation of Off-line Handwritten Documents Chen Huang and Sargur N. Srihari {chuang5, srihari}@cedar.buffalo.edu Center of Excellence for Document Analysis and Recognition (CEDAR), Department
More informationIntroduction to Modeling and Simulation. Conceptual Modeling. OSMAN BALCI Professor
Introduction to Modeling and Simulation Conceptual Modeling OSMAN BALCI Professor Department of Computer Science Virginia Polytechnic Institute and State University (Virginia Tech) Blacksburg, VA 24061,
More informationTwitter Sentiment Classification on Sanders Data using Hybrid Approach
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661,p-ISSN: 2278-8727, Volume 17, Issue 4, Ver. I (July Aug. 2015), PP 118-123 www.iosrjournals.org Twitter Sentiment Classification on Sanders
More informationLinking Task: Identifying authors and book titles in verbose queries
Linking Task: Identifying authors and book titles in verbose queries Anaïs Ollagnier, Sébastien Fournier, and Patrice Bellot Aix-Marseille University, CNRS, ENSAM, University of Toulon, LSIS UMR 7296,
More informationOPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS
OPTIMIZATINON OF TRAINING SETS FOR HEBBIAN-LEARNING- BASED CLASSIFIERS Václav Kocian, Eva Volná, Michal Janošek, Martin Kotyrba University of Ostrava Department of Informatics and Computers Dvořákova 7,
More informationMatching Similarity for Keyword-Based Clustering
Matching Similarity for Keyword-Based Clustering Mohammad Rezaei and Pasi Fränti University of Eastern Finland {rezaei,franti}@cs.uef.fi Abstract. Semantic clustering of objects such as documents, web
More informationEdX Learner s Guide. Release
EdX Learner s Guide Release Nov 18, 2017 Contents 1 Welcome! 1 1.1 Learning in a MOOC........................................... 1 1.2 If You Have Questions As You Take a Course..............................
More informationZACHARY J. OSTER CURRICULUM VITAE
ZACHARY J. OSTER CURRICULUM VITAE McGraw Hall 108 Phone: (262) 472-5006 800 W. Main St. Email: osterz@uww.edu Whitewater, WI 53190 Website: http://cs.uww.edu/~osterz/ RESEARCH INTERESTS Formal methods
More informationLaboratorio di Intelligenza Artificiale e Robotica
Laboratorio di Intelligenza Artificiale e Robotica A.A. 2008-2009 Outline 2 Machine Learning Unsupervised Learning Supervised Learning Reinforcement Learning Genetic Algorithms Genetics-Based Machine Learning
More informationCOURSE INFORMATION. Course Number SER 216. Course Title Software Enterprise II: Testing and Quality. Credits 3. Prerequisites SER 215
**Disclaimer** This syllabus is to be used as a guideline only. The information provided is a summary of topics to be covered in the class. Information contained in this document such as assignments, grading
More informationA Case-Based Approach To Imitation Learning in Robotic Agents
A Case-Based Approach To Imitation Learning in Robotic Agents Tesca Fitzgerald, Ashok Goel School of Interactive Computing Georgia Institute of Technology, Atlanta, GA 30332, USA {tesca.fitzgerald,goel}@cc.gatech.edu
More informationOCR LEVEL 3 CAMBRIDGE TECHNICAL
Cambridge TECHNICALS OCR LEVEL 3 CAMBRIDGE TECHNICAL CERTIFICATE/DIPLOMA IN IT SYSTEMS ANALYSIS K/505/5481 LEVEL 3 UNIT 34 GUIDED LEARNING HOURS: 60 UNIT CREDIT VALUE: 10 SYSTEMS ANALYSIS K/505/5481 LEVEL
More informationEECS 571 PRINCIPLES OF REAL-TIME COMPUTING Fall 10. Instructor: Kang G. Shin, 4605 CSE, ;
EECS 571 PRINCIPLES OF REAL-TIME COMPUTING Fall 10 Instructor: Kang G. Shin, 4605 CSE, 763-0391; kgshin@umich.edu Number of credit hours: 4 Class meeting time and room: Regular classes: MW 10:30am noon
More informationCS Machine Learning
CS 478 - Machine Learning Projects Data Representation Basic testing and evaluation schemes CS 478 Data and Testing 1 Programming Issues l Program in any platform you want l Realize that you will be doing
More informationMaximizing Learning Through Course Alignment and Experience with Different Types of Knowledge
Innov High Educ (2009) 34:93 103 DOI 10.1007/s10755-009-9095-2 Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Phyllis Blumberg Published online: 3 February
More informationEducation the telstra BLuEPRint
Education THE TELSTRA BLUEPRINT A quality Education for every child A supportive environment for every teacher And inspirational technology for every budget. is it too much to ask? We don t think so. New
More informationEntrepreneurial Discovery and the Demmert/Klein Experiment: Additional Evidence from Germany
Entrepreneurial Discovery and the Demmert/Klein Experiment: Additional Evidence from Germany Jana Kitzmann and Dirk Schiereck, Endowed Chair for Banking and Finance, EUROPEAN BUSINESS SCHOOL, International
More informationPython Machine Learning
Python Machine Learning Unlock deeper insights into machine learning with this vital guide to cuttingedge predictive analytics Sebastian Raschka [ PUBLISHING 1 open source I community experience distilled
More informationSchool Inspection in Hesse/Germany
Hessisches Kultusministerium School Inspection in Hesse/Germany Contents 1. Introduction...2 2. School inspection as a Procedure for Quality Assurance and Quality Enhancement...2 3. The Hessian framework
More informationProgramme Specification. MSc in International Real Estate
Programme Specification MSc in International Real Estate IRE GUIDE OCTOBER 2014 ROYAL AGRICULTURAL UNIVERSITY, CIRENCESTER PROGRAMME SPECIFICATION MSc International Real Estate NB The information contained
More informationGuidelines for Project I Delivery and Assessment Department of Industrial and Mechanical Engineering Lebanese American University
Guidelines for Project I Delivery and Assessment Department of Industrial and Mechanical Engineering Lebanese American University Approved: July 6, 2009 Amended: July 28, 2009 Amended: October 30, 2009
More informationA student diagnosing and evaluation system for laboratory-based academic exercises
A student diagnosing and evaluation system for laboratory-based academic exercises Maria Samarakou, Emmanouil Fylladitakis and Pantelis Prentakis Technological Educational Institute (T.E.I.) of Athens
More informationIndiana Collaborative for Project Based Learning. PBL Certification Process
Indiana Collaborative for Project Based Learning ICPBL Certification mission is to PBL Certification Process ICPBL Processing Center c/o CELL 1400 East Hanna Avenue Indianapolis, IN 46227 (317) 791-5702
More informationEDIT 576 (2 credits) Mobile Learning and Applications Fall Semester 2015 August 31 October 18, 2015 Fully Online Course
GEORGE MASON UNIVERSITY COLLEGE OF EDUCATION AND HUMAN DEVELOPMENT INSTRUCTIONAL DESIGN AND TECHNOLOGY PROGRAM EDIT 576 (2 credits) Mobile Learning and Applications Fall Semester 2015 August 31 October
More informationProduct Feature-based Ratings foropinionsummarization of E-Commerce Feedback Comments
Product Feature-based Ratings foropinionsummarization of E-Commerce Feedback Comments Vijayshri Ramkrishna Ingale PG Student, Department of Computer Engineering JSPM s Imperial College of Engineering &
More informationImplementing a tool to Support KAOS-Beta Process Model Using EPF
Implementing a tool to Support KAOS-Beta Process Model Using EPF Malihe Tabatabaie Malihe.Tabatabaie@cs.york.ac.uk Department of Computer Science The University of York United Kingdom Eclipse Process Framework
More informationExperiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling
Experiments with SMS Translation and Stochastic Gradient Descent in Spanish Text Author Profiling Notebook for PAN at CLEF 2013 Andrés Alfonso Caurcel Díaz 1 and José María Gómez Hidalgo 2 1 Universidad
More informationTest Effort Estimation Using Neural Network
J. Software Engineering & Applications, 2010, 3: 331-340 doi:10.4236/jsea.2010.34038 Published Online April 2010 (http://www.scirp.org/journal/jsea) 331 Chintala Abhishek*, Veginati Pavan Kumar, Harish
More informationExploration. CS : Deep Reinforcement Learning Sergey Levine
Exploration CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 4 due on Wednesday 2. Project proposal feedback sent Today s Lecture 1. What is exploration? Why is it a problem?
More information*Net Perceptions, Inc West 78th Street Suite 300 Minneapolis, MN
From: AAAI Technical Report WS-98-08. Compilation copyright 1998, AAAI (www.aaai.org). All rights reserved. Recommender Systems: A GroupLens Perspective Joseph A. Konstan *t, John Riedl *t, AI Borchers,
More informationGuidelines for the Use of the Continuing Education Unit (CEU)
Guidelines for the Use of the Continuing Education Unit (CEU) The UNC Policy Manual The essential educational mission of the University is augmented through a broad range of activities generally categorized
More informationLeveraging MOOCs to bring entrepreneurship and innovation to everyone on campus
Paper ID #9305 Leveraging MOOCs to bring entrepreneurship and innovation to everyone on campus Dr. James V Green, University of Maryland, College Park Dr. James V. Green leads the education activities
More informationre An Interactive web based tool for sorting textbook images prior to adaptation to accessible format: Year 1 Final Report
to Anh Bui, DIAGRAM Center from Steve Landau, Touch Graphics, Inc. re An Interactive web based tool for sorting textbook images prior to adaptation to accessible format: Year 1 Final Report date 8 May
More informationNovember 17, 2017 ARIZONA STATE UNIVERSITY. ADDENDUM 3 RFP Digital Integrated Enrollment Support for Students
November 17, 2017 ARIZONA STATE UNIVERSITY ADDENDUM 3 RFP 331801 Digital Integrated Enrollment Support for Students Please note the following answers to questions that were asked prior to the deadline
More informationActivities, Exercises, Assignments Copyright 2009 Cem Kaner 1
Patterns of activities, iti exercises and assignments Workshop on Teaching Software Testing January 31, 2009 Cem Kaner, J.D., Ph.D. kaner@kaner.com Professor of Software Engineering Florida Institute of
More informationSwitchboard Language Model Improvement with Conversational Data from Gigaword
Katholieke Universiteit Leuven Faculty of Engineering Master in Artificial Intelligence (MAI) Speech and Language Technology (SLT) Switchboard Language Model Improvement with Conversational Data from Gigaword
More informationCOURSE LISTING. Courses Listed. Training for Cloud with SAP SuccessFactors in Integration. 23 November 2017 (08:13 GMT) Beginner.
Training for Cloud with SAP SuccessFactors in Integration Courses Listed Beginner SAPHR - SAP ERP Human Capital Management Overview SAPHRE - SAP ERP HCM Overview Advanced HRH00E - SAP HCM/SAP SuccessFactors
More informationAustralian Journal of Basic and Applied Sciences
AENSI Journals Australian Journal of Basic and Applied Sciences ISSN:1991-8178 Journal home page: www.ajbasweb.com Feature Selection Technique Using Principal Component Analysis For Improving Fuzzy C-Mean
More informationIdentification of Opinion Leaders Using Text Mining Technique in Virtual Community
Identification of Opinion Leaders Using Text Mining Technique in Virtual Community Chihli Hung Department of Information Management Chung Yuan Christian University Taiwan 32023, R.O.C. chihli@cycu.edu.tw
More informationExperiment Databases: Towards an Improved Experimental Methodology in Machine Learning
Experiment Databases: Towards an Improved Experimental Methodology in Machine Learning Hendrik Blockeel and Joaquin Vanschoren Computer Science Dept., K.U.Leuven, Celestijnenlaan 200A, 3001 Leuven, Belgium
More informationDeploying Agile Practices in Organizations: A Case Study
Copyright: EuroSPI 2005, Will be presented at 9-11 November, Budapest, Hungary Deploying Agile Practices in Organizations: A Case Study Minna Pikkarainen 1, Outi Salo 1, and Jari Still 2 1 VTT Technical
More informationP. Belsis, C. Sgouropoulou, K. Sfikas, G. Pantziou, C. Skourlas, J. Varnas
Exploiting Distance Learning Methods and Multimediaenhanced instructional content to support IT Curricula in Greek Technological Educational Institutes P. Belsis, C. Sgouropoulou, K. Sfikas, G. Pantziou,
More informationHistorical maintenance relevant information roadmap for a self-learning maintenance prediction procedural approach
IOP Conference Series: Materials Science and Engineering PAPER OPEN ACCESS Historical maintenance relevant information roadmap for a self-learning maintenance prediction procedural approach To cite this
More informationFinancing Education In Minnesota
Financing Education In Minnesota 2016-2017 Created with Tagul.com A Publication of the Minnesota House of Representatives Fiscal Analysis Department August 2016 Financing Education in Minnesota 2016-17
More informationRequirements-Gathering Collaborative Networks in Distributed Software Projects
Requirements-Gathering Collaborative Networks in Distributed Software Projects Paula Laurent and Jane Cleland-Huang Systems and Requirements Engineering Center DePaul University {plaurent, jhuang}@cs.depaul.edu
More informationSARDNET: A Self-Organizing Feature Map for Sequences
SARDNET: A Self-Organizing Feature Map for Sequences Daniel L. James and Risto Miikkulainen Department of Computer Sciences The University of Texas at Austin Austin, TX 78712 dljames,risto~cs.utexas.edu
More informationAQUA: An Ontology-Driven Question Answering System
AQUA: An Ontology-Driven Question Answering System Maria Vargas-Vera, Enrico Motta and John Domingue Knowledge Media Institute (KMI) The Open University, Walton Hall, Milton Keynes, MK7 6AA, United Kingdom.
More informationLaboratorio di Intelligenza Artificiale e Robotica
Laboratorio di Intelligenza Artificiale e Robotica A.A. 2008-2009 Outline 2 Machine Learning Unsupervised Learning Supervised Learning Reinforcement Learning Genetic Algorithms Genetics-Based Machine Learning
More informationADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY DOWNLOAD EBOOK : ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY PDF
Read Online and Download Ebook ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY DOWNLOAD EBOOK : ADVANCED MACHINE LEARNING WITH PYTHON BY JOHN HEARTY PDF Click link bellow and free register to download
More informationLearning Optimal Dialogue Strategies: A Case Study of a Spoken Dialogue Agent for
Learning Optimal Dialogue Strategies: A Case Study of a Spoken Dialogue Agent for Email Marilyn A. Walker Jeanne C. Fromer Shrikanth Narayanan walker@research.att.com jeannie@ai.mit.edu shri@research.att.com
More informationSeminar - Organic Computing
Seminar - Organic Computing Self-Organisation of OC-Systems Markus Franke 25.01.2006 Typeset by FoilTEX Timetable 1. Overview 2. Characteristics of SO-Systems 3. Concern with Nature 4. Design-Concepts
More informationSoftware Development Plan
Version 2.0e Software Development Plan Tom Welch, CPC Copyright 1997-2001, Tom Welch, CPC Page 1 COVER Date Project Name Project Manager Contact Info Document # Revision Level Label Business Confidential
More informationSt. Mary Cathedral Parish & School
Parish School Governance St. Mary Cathedral Parish & School School Advisory Council Constitution Approved by Parish Pastoral Council April 25, 2014 -i- Constitution of the St. Mary Cathedral School Advisory
More informationNottingham Trent University Course Specification
Nottingham Trent University Course Specification Basic Course Information 1. Awarding Institution: Nottingham Trent University 2. School/Campus: Nottingham Business School / City 3. Final Award, Course
More informationMining Association Rules in Student s Assessment Data
www.ijcsi.org 211 Mining Association Rules in Student s Assessment Data Dr. Varun Kumar 1, Anupama Chadha 2 1 Department of Computer Science and Engineering, MVN University Palwal, Haryana, India 2 Anupama
More informationIntroduction to Causal Inference. Problem Set 1. Required Problems
Introduction to Causal Inference Problem Set 1 Professor: Teppei Yamamoto Due Friday, July 15 (at beginning of class) Only the required problems are due on the above date. The optional problems will not
More informationPreference Learning in Recommender Systems
Preference Learning in Recommender Systems Marco de Gemmis, Leo Iaquinta, Pasquale Lops, Cataldo Musto, Fedelucio Narducci, and Giovanni Semeraro Department of Computer Science University of Bari Aldo
More informationCOMPUTER-ASSISTED INDEPENDENT STUDY IN MULTIVARIATE CALCULUS
COMPUTER-ASSISTED INDEPENDENT STUDY IN MULTIVARIATE CALCULUS L. Descalço 1, Paula Carvalho 1, J.P. Cruz 1, Paula Oliveira 1, Dina Seabra 2 1 Departamento de Matemática, Universidade de Aveiro (PORTUGAL)
More informationThe Use of Statistical, Computational and Modelling Tools in Higher Learning Institutions: A Case Study of the University of Dodoma
International Journal of Computer Applications (975 8887) The Use of Statistical, Computational and Modelling Tools in Higher Learning Institutions: A Case Study of the University of Dodoma Gilbert M.
More informationOn-the-Fly Customization of Automated Essay Scoring
Research Report On-the-Fly Customization of Automated Essay Scoring Yigal Attali Research & Development December 2007 RR-07-42 On-the-Fly Customization of Automated Essay Scoring Yigal Attali ETS, Princeton,
More informationNew Venture Financing
New Venture Financing General Course Information: FINC-GB.3373.01-F2017 NEW VENTURE FINANCING Tuesdays/Thursday 1.30-2.50pm Room: TBC Course Overview and Objectives This is a capstone course focusing on
More informationTesting A Moving Target: How Do We Test Machine Learning Systems? Peter Varhol Technology Strategy Research, USA
Testing A Moving Target: How Do We Test Machine Learning Systems? Peter Varhol Technology Strategy Research, USA Testing a Moving Target How Do We Test Machine Learning Systems? Peter Varhol, Technology
More informationlearning collegiate assessment]
[ collegiate learning assessment] INSTITUTIONAL REPORT 2005 2006 Kalamazoo College council for aid to education 215 lexington avenue floor 21 new york new york 10016-6023 p 212.217.0700 f 212.661.9766
More informationCHAPTER 4: REIMBURSEMENT STRATEGIES 24
CHAPTER 4: REIMBURSEMENT STRATEGIES 24 INTRODUCTION Once state level policymakers have decided to implement and pay for CSR, one issue they face is simply how to calculate the reimbursements to districts
More informationLongest Common Subsequence: A Method for Automatic Evaluation of Handwritten Essays
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661,p-ISSN: 2278-8727, Volume 17, Issue 6, Ver. IV (Nov Dec. 2015), PP 01-07 www.iosrjournals.org Longest Common Subsequence: A Method for
More informationUSER ADAPTATION IN E-LEARNING ENVIRONMENTS
USER ADAPTATION IN E-LEARNING ENVIRONMENTS Paraskevi Tzouveli Image, Video and Multimedia Systems Laboratory School of Electrical and Computer Engineering National Technical University of Athens tpar@image.
More information