USING INTELLIGENT TUTORS TO ENHANCE STUDENT LEARNING OF APPLICATION PROGRAMMING INTERFACES Aniket Dahotre, Vasanth Krishnamoorthy, Matt Corley, Christopher Scaffidi Oregon State University, Corvallis, OR 97330 {dahotrea, krishnav, corleym, cscaffid}@onid.orst.edu ABSTRACT An essential part of software engineering training is for students to learn how to effectively use application programming interfaces (APIs), but traditional instruction only provides direct support for helping students to learn the most commonly used APIs. This paper introduces a new approach whereby professors could delegate some of these training responsibilities to intelligent tutors, which are interactive instructional materials that tailor themselves to each student s progress. A prototype system has been developed that semi-automatically generates API tutors from open source code freely available on the web. As API tutors are published to a new website, students have an increasingly large menu of training materials available for them to choose from. A preliminary study indicates that the approach increases student learning on sample tasks.
INTRODUCTION Successful software engineering depends critically on an in-depth knowledge of how to combine high-level building blocks, particularly application programming interfaces (APIs), which provide the abstractions for many important functions such as reading data from files and displaying web pages inside of software applications. Learning APIs is not always easy. Researchers have identified many ways in which APIs can confuse people [17,18] and reduce their productivity by as much as a factor of 6 on typical programming tasks [9,18,20]. However, lectures, textbooks, and other traditional instruction techniques only expose students to the most commonly used APIs. For example, the standard Java API currently includes over 34,000 methods and classes, while the Microsoft.NET API has over 140,000 [19]. When students graduate and move into more specialized areas of software engineering, they have relatively little direct experience with most specialized APIs. As a result, novice software engineers learn many specialized APIs while struggling over their first few years as software engineers [4]. This paper contributes a new strategy for training software engineering students by introducing an approach whereby professors could delegate some training responsibilities to intelligent tutors, which are interactive instructional materials that tailor themselves to each student s progress [11]. Professors would continue to teach the most commonly-used APIs, while students interested in particular specializations would be provided with opportunities to learn specialized APIs from intelligent tutors. These intelligent API tutors, in turn, would be created semi-automatically by professors teaching assistants using example code downloaded from the web, and the tutors would
be published to a website so that students can use them remotely. The ultimate vision is a day when instructors from multiple universities pool their API tutors into a common website containing tens of thousands of API tutors. Thus, API tutors effectively would serve as a mediating mechanism for enabling instructors to provide students with an extensive menu of training materials to use for learning APIs. This paper describes a first step toward this vision, presenting a prototype system and preliminary experiment showing that intelligent API tutors increase student learning in a laboratory setting. This result lays the groundwork for future research aimed at enabling students to learn a broad range of specialized APIs on demand in classroom, distance-education, and on-the-job settings. RELATED WORK When novices struggle with learning an API, they usually try to cope by looking for code examples that show how to properly use the API [4,5]. The approach is favored particularly in the frequent case where API documentation is incomprehensible, incomplete, or outright missing [17]. A succession of increasingly powerful tools help to augment documentation, or even to automate the frequent process of finding, copying, and pasting code (e.g., [19]). These systems do an effective job of retrieving examples for a given topic and even finding the most common way in which an API is used for a particular task. Yet this search-copy-paste process strips the examples of any comments, context, and explanatory prose that had been available to explain why certain API calls are needed [5]. The result is that people using this approach may lack a thorough
understanding of APIs demonstrated by the code [13]. This can lead to trusting code that actually has errors [5] or combining APIs incorrectly [17]. Thus, effective approaches are needed to help people not just to obtain code examples, but also to learn from them. A promising approach is to apply intelligent tutors, which are interactive instructional materials that teach students how to perform procedural tasks and that tailor themselves to each student s progress [3,11]. An example-tracing intelligent tutor contains multiple screens or windows, each of which provides a problem-solving example, explanatory prose, or a quiz containing a problem that tests student knowledge of how to solve a problem similar to examples previously provided to the student. One of the most well-developed systems for creating intelligent tutors is the Cognitive Tutor Authoring Tools (CTAT) suite [2]. With CTAT, a teacher can create a screen by simply dragging and dropping labels or textboxes into place (much like designing a presentation in Microsoft PowerPoint). For a quiz problem, the teacher then demonstrates valid and invalid solution methods; from these examples, CTAT automatically generates a generalized example that serves as a model for determining if a student needs hints. CTAT combines the examples and quizzes together into a Java or Flash program that can then be viewed by students through a website. This program constitutes the intelligent tutor. As each student works through an intelligent tutor s examples and quizzes, the tutor reports data back to the website so that additional tutors can be selected for this student, in order to build on strengths and ameliorate weaknesses. Studies consistently show that compared to having students simply learn from textbooks and lectures, intelligent tutors increase the amount that students learn in a given amount of
time [8,14]. They have proven effective in teaching introductory programming concepts including loops, records, and lists [3,6,10,15,16]. To date, however, intelligent tutors have not been applied to the problem of teaching people how to use APIs. APPROACH A new prototype tool suite called ApiTutor has been developed by extending CTAT. ApiTutor mines examples of Java API usage from the web, in order to help teachers with creating intelligent tutors (Figure 1). The teacher begins by entering keywords such as connect to database ; ApiTutor appends the words java examples and forwards the resulting query to a set of existing search engines (currently Yahoo and Bing). The search engines return a list of web pages. As described in detail below, ApiTutor retrieves these pages, then extracts and intelligently selects a set of code examples presented in those pages. It provides a user interface whereby the teacher can organize these examples into screens comprising a new intelligent tutor. Within the intelligent tutor, some of these examples are typically used for teaching, while others are used for fill-in-the-blank programming quizzes. Finally, ApiTutor generates the intelligent tutor as a Java program, which it publishes to a website for students to view.
2. ApiTutor uses keywords to retrieve web pages about topic Search engines Teacher 1. Teacher enters topic keywords 4. ApiTutor presents examples to teacher for further filtering, organizing, and annotation with hints Teacher tool 5. ApiTutor publishes intelligent tutor 3. ApiTutor identifies, parses, compares, and filters code examples Website Figure 1. Procedure for creating and publishing an intelligent tutor with the ApiTutor prototype. This system includes a tool that the teacher directly uses to create an intelligent tutor, as well as a website where intelligent tutors are published. Extracting code examples: Since web pages about APIs contain various types of textual and multimedia elements in addition to code samples, ApiTutor must extract code samples that can be utilized for generating intelligent tutors. This is straightforward, since most code samples are enclosed in either pre or code HTML tags; for example, 85% of the code samples in the top 10 search results for connect to database and how to use hashmap are enclosed in one of these tags. Consequently, to extract code snippets from web pages, ApiTutors identifies text enclosed by these tags to retrieve a list of candidate code examples. It attempts to parse each example using a grammar for the Java language, and it discards any code example that cannot be parsed into a valid compilation unit (e.g., a statement, basic block, method, or class). The system continues retrieving and parsing examples until obtaining at least 15 code examples (which is a configurable threshold).
Finding similar code examples: To find the most common way of using an API, ApiTutor compares the code examples and ranks them based on mutual similarity. This is accomplished by extracting the following items (called terms below) from each code example s parse tree: the names of packages imported, the names of classes referenced, the names of methods invoked, and the list of classes for variables instantiated. Using these terms, ApiTutor converts each code example to a TF-IDF representation [7], which is the same underlying representation used in many other systems that search the web for source code or that compute the similarity of two pieces of source code (e.g., [12]). This conversion to TF-IDF represents an example x as a vector <x 1,x 2,...,x R >, where there are R unique terms (over all examples) and for each term t, x t = # of occurrences of t in x # of occurrences of u in x) u log total # of examples # of examples that contain t The system then computes the dot product between each pair of vectors, yielding a matrix representing similarities among code examples (where each entry is between 0 and 1). Finally, ApiTutor uses the largest entries in this matrix to sort the examples from most to least similar. As a result, the examples that are brought first to the teacher s attention are those that have the highest mutual similarity. Organizing examples into an intelligent tutor: ApiTutors presents the code examples to the teacher for organization into an intelligent tutor. The teacher can indicate that each code example should be used to generate a quiz, used to generate a screen showing the example (with teacher-specified explanatory prose), or discarded. For examples that will be turned into a quiz, the teacher can select terms that should be
blanked out in other words, these are the names of methods, classes, or packages that the student will need to provide in order to complete the quiz. This form of quiz is essentially identical to the commonly-used kind of exam question where students must fill in a code skeleton to create a correct program. Typically, the first few screens of an intelligent tutor will give examples, then quizzes with only a few names blanked out, then more complex examples, and finally quizzes with many names blanked out. For each blanked-out field in a quiz, the teacher can enter hints. The teacher also provides a difficulty rating for each example and quiz, so that these screens can be sorted in order of increasing difficulty. Generating the final intelligent tutor: Based on these pieces of information from the teacher, ApiTutor uses CTAT to actually generate the specified intelligent tutor. The tutor is posted to a website, where students can download and view it. As each student works through an intelligent tutor, hints are displayed if the student is unable to fill in the blanks of quizzes. If the student answers the quiz correctly without receiving hints, then the tutor progresses to examples and quizzes of increasing difficulty. If the student requires hints, then examples and quizzes of comparable difficulty are displayed next. If the student fails the quiz, then the tutor temporarily returns to easier materials before returning to the quiz. PRELIMINARY EMPIRICAL EVALUATION A preliminary laboratory experiment has been conducted to evaluate how well ApiTutor enhances student learning of sample APIs compared to traditional textbook-
based training. Ten computer science students (undergraduate or master s) were recruited; each was assigned two randomly-selected tasks. Each task had two parts. In each task s first part, each student learned about an API from a textbook or from an intelligent tutor. This training mode was randomly assigned, and the API was randomly selected from a list of 5 Java APIs covered in undergraduate courses (specifically related to the StringBuffer, FileWriter, HashMap, XML DocumentBuilder, and JDBC Connection classes). Second, each task concluded with a three-question fill-in-the-blank quiz on paper, similar to what would be expected on a midterm or final exam. To maximize comparability between the two training modes, students had the same amount of time, 15 minutes, to read the API s training materials (but were allowed to move on to the quiz earlier if desired). Regardless of training mode, students were verbally encouraged to focus on class and method names, and they were told that they would be tested on remembering these names. Moreover, each student was provided at the start of the experiment with an example of a quiz so they would be prepared for a fill-in-theblank quiz. Finally, in both training modes, students learned entirely from training materials, with no human tutoring at all. Each participant received $10 in compensation. Averaged over all five APIs, students scored 93% after ApiTutor-based training and scored only 45% after textbook-based training, with an effect size of 1.8 (equal to that of the best intelligent tutors in related work [8]). A two-factor ANOVA indicated that both the training mode and the API topic had a significant effect on student scores (P<0.01), with no interaction between these (at P<0.10), indicating that ApiTutor was approximately equally effective on all APIs tested. It turned out that in the textbook
training mode, students always chose to use the full 15 minutes allowed for studying the API; in contrast, they chose to use an average of only 10 minutes in the ApiTutor training mode before asking to start the quiz. Consequently, compared to textbook-based training, ApiTutor training enabled students to score higher while using less time for training. FUTURE WORK Future research will go beyond this preliminary experiment by evaluating the effectiveness of the approach in real-world settings. In particular, this work will evaluate how well teaching assistants outside the research team are able to use ApiTutors to create intelligent tutors for a much broader range of languages and APIs, including many complex, specialized APIs that cannot be mastered in just a few minutes. Future work will include assessing how much lecture or introductory teaching is required before students can use APIs; for example, students would presumably need to hear about the concept of database tuples before they could learn to use the JDBC API, regardless of the specific API training mode used. In addition, future work will evaluate how well intelligent tutors created by teaching assistants are able to increase student learning in the laboratory, in undergraduate courses, in distance-learning courses, and in on-the-job training. It would be especially worthwhile to extend this approach to help new employees learn proprietary in-house APIs used by their companies, since adequate training materials are often lacking for this purpose.
REFERENCES [1] ACM Java Task Force Report, Association for Computing Machinery, NY, 2006. [2] Aleven, V, McLaren, B, Sewall, J, and Koedinger, K. Example-tracing tutors: A new paradigm for intelligent tutoring systems. International Journal of Artificial Intelligence and Education. 19, 105-154, 2009. [3] Anderson, J, Boyle, C, Corbett, A, and Lewis, M. Cognitive modeling and intelligent tutoring. Artificial Intelligence and Learning Environments, 4-21, 1990. [4] Begel, A, and Simon, B. Struggles of new college graduates in their first software development job. 39th SIGCSE Technical Symposium on Computer Science Education, 226-230, 2008. [5] Brandt, J, Guo, P, Lewenstein, J, Dontcheva, M, and Klemmer, S. Two studies of opportunistic programming: Interleaving web foraging, learning, and writing code. 27th International Conference on Human Factors in Computing Systems, 1589-1598, 2009. [6] Butz, C, Hua, S, and Maguire, R. A web-based intelligent tutoring system for computer programming. IEEE/WIC/ACM International Conference on Web Intelligence, 159-165, 2005. [7] Chakrabarti, S. Mining the Web: Discovering Knowledge from Hypertext Data, Morgan Kaufmann, 2002. [8] Corbett, A. Cognitive computer tutors: Solving the two-sigma problem. User Modeling 2001, 137-147, 2001.
[9] Ellis, B, Stylos, J, and Myers, B. The factory pattern in API design: A usability evaluation. 29th International Conference on Software Engineering, 302-312, 2007. [10] Holland, J, Mitrovic, A, and Martin, B. J-LATTE: A constraint-based tutor for Java. 17th International on Conference on Computers in Education, 142-146, 2009. [11] Koedinger, K, Anderson, J, Hadley, W, and Mark, M. Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education. 8(1), 30-43, 1997. [12] Kuhn, A, Ducasse, S, and Girba, T. Semantic clustering: Identifying topics in source code. Information and Software Technology. 49(3), 230-243, 2007. [13] McLellan, S, Roesler, A, Tempest, J, and Spinuzzi, C. Building more usable APIs. IEEE Software. 15(3), 78-86, 2002. [14] Merrill, D, Reiser, B, Ranney, M, and Trafton, J. Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. Journal of the Learning Sciences. 2(3), 277-305, 1992. [15] Mitrovic, A, and Ohlsson, S. Evaluation of a constraint-based tutor for a database. International Journal of Artificial Intelligence in Education. 10(3-4), 238-256, 1999. [16] Reiser, B, Anderson, J, and Farrell, R. Dynamic student modelling in an intelligent tutor for Lisp programming. 9th International Joint Conference on Artificial Intelligence, 8-13, 1985.
[17] Robillard, M, and DeLine, R. A field study of API learning obstacles. Empirical Software Engineering. 15, 1-30, 2010. [18] Stylos, J, and Clarke, S. Usability implications of requiring parameters in objects constructors. 29th International Conference on Software Engineering, 529-539, 2007. [19] Stylos, J, and Myers, B. Mica: A web-search tool for finding API components and examples. IEEE Symposium on Visual Languages and Human-Centric Computing, 195-202, 2006. [20] Stylos, J, Graf, B, Busse, D, Ziegler, C, Ehret, R, and Karstens, J. A case study of API redesign for improved usability. IEEE Symposium on Visual Languages and Human-Centric Computing, 189-192, 2008.