Categorizing & Discovering Course Information for BU Abroad Programs A Usability Study by Andrew Bauer Website Overview The website I selected is that of Boston University s Study Abroad department. I chose this site largely because of its extensive and particular content structure. The site offers an in-depth look into all programs offered (worldwide) by the department. With over a fluid list of over a hundred programs, a flat structure would be far from optimal.
Traditionally especially in WordPress (this site s CMS) content is sorted into a single, clear category/subcategory structure. While that was certainly a potential choice here, for this implementation it would be far too limiting. Which categories would be primary? Would we group things by Location first, or by Focus Area? Would our classification of content match up with users expectations? Instead of using Categories, the site took advantage of Custom Taxonomies in WordPress. Custom Taxonomies allow content to be grouped together by topic, broken down by custom attributes. In this sense, Topic: French, Topic: Social Studies, Location: France, Credits: Yes could all be applied to a single listing. These attributes can then be used to search, filter, and display an incredibly refined interface. http://www.bu.edu/abroad/ Evaluation Purpose & Use Case Scenarios The timing of this assignment was incredibly serendipitous; I happened to be working on a project to update the handling of Taxonomies in BU s WordPress installation while looking for a topic to cover here. A key part of understanding Comment [MS1]: excellent Taxonomies is realizing how content is (1) consumed (from user s side), and (2) organized and administered (from the Site Owner s side.) Without a thorough understanding of both sides, any development work chances being at best unhelpful or at worst, entirely destructive.
Since my current development work focused entirely on the back-end functionality, my primary research was to ensure I hadn t broken anything there. Before this study, I conducted informal discussions with content managers to determine common practices and most-valued features. Using these insights, I began development of the updates. The evaluation purpose of this study is to analyze information usage relating to the storage/publication of course information (by content managers) and its consumption (by users). For this study I set two parallel, but related goals: 1) Evaluate new Taxonomy interface in relation to user (content manager) goals, and 2) Evaluate users (visitors ) process for finding information within the current site. Comment [MS2]: Even though you detail these cases below, I think the reader would benefit here from more detail about the tasks involved. Use Case #1: Content Managers Updating courses: Assigning relevant attributes. A primary task of content managers is adding and updating online listings of courses. Grouping related courses requires assigning taxonomy terms (e.g. Focus -> French Studies) to a course listing. These groupings are critical for both site users (front-end) and content managers (backend.) On the front end, the theme displays together all course matching specific criteria (e.g. Courses in France.) On the back end, tagging listings consistent with the site s IA allows (other) content managers to efficiently find and update content. Previously, content managers assigned terms by checking boxes from a single (long) list. The version tested consists of an interface with three tabs: Assigned, Most
Used, and View All. The tested version also includes a quick search box, allowing users to search for the desired taxonomy term. Within the use case, this study evaluates the speed and accuracy of assigning terms to the current post, with the new interface. Factors also considered in this study are users order of operations, design feedback, functional feedback, and miscellaneous comments. Use Case #2: End-Users (Site Visitors) Research: Learning more about BU Abroad offerings. For site visitors, a primary task is learning about the programs offered by BU Abroad. The site has been out in the wild for some time, but has never been formally tested. I was curious to measure the efficiency of the information hierarchy for end-users. This use case allows for assessment of visitors efficiency at discovering information with the current theme. The goal of testing this use case is to determine successful and weak elements of the front-end design. A strong link to back-end content structure is also present here, directly affecting the front-end user experience. Recruitment For this study, I recruited participants that were employees of Boston University and worked in Information Technology. Other fixed criteria included: (1) one participant must have little-to-no experience with WordPress, (2) one participant must have experience working with WordPress AND limited experience with Taxonomies, and (3)
one participant must be a content manager, with WordPress and Taxonomies experience. Aside from the above criteria, all other participant demographics were random. The study included one female and two male participants, with ages ranging from twenty-three to fifty-six years old. Other demographic information was not tracked. Comment [MS3]: Okay this is good, but I actually thought, until I skimmed ahead, that you were going to have two different sets of users to test since you are evaluating tasks that are oriented toward different interfaces (the backend of Wordpress vs. the front-end of the website). I would think a proper evaluation of the second task could benefit from another set of participants. Comment [MS4]: How did you go about soliciting their help? Data Collection Data was collected both formally with the Morae Manager and informally, through verbal feedback. The formal testing took place on an Apple MacBook Pro, Comment [MS5]: What do you mean by this? Just because it is verbal does not make it informal running Windows 7 in a full-screen VMWare virtual machine. To prepare the environment for evaluation, the Morae Recorder software was installed. Additionally, links to the BU Abroad homepage (for Task 1) and the Grenoble French Studies Program page (for Task 2) were placed on the Desktop. Firefox was used as the web browser for this study. Informal evaluation note taking of verbal feedback was recorded by hand. To begin evaluation, participants were asked to provide verbal consent, informed that the screen was being recorded, asked if they had any questions, and instructed to click the appropriate link on the Desktop to launch the web browser. After the web Comment [MS6]: Great browser was launched, participants were verbally given the instructions listed below. Other than clarification of the instructions, no assistance was provided.
Task 1: (Front-end) Research Course Information 1. Find the 2014 tuition for the French Studies Program in Grenoble, France. 2. Find the number programs available with a focus of Physics. Task 2: (Admin) Update Course Listings 1. The Grenoble French Studies Program is no longer available in France, but is now available in London and Argentina. Edit the page & update the Locations box with this change. 2. The Grenoble French Studies Program is expanding! Biology is now a new (additional) focus of the program. Edit the page & update the Foci box with this change. Comment [MS7]: Ah, good! THIS is what I had expected to see with the use-case scenario. Analysis In the Morae application, tracked metrics included task duration, task completion rate, error rate, and a categorical determination of task completion quality. This data was supplemented with qualitative findings of the user experience, based on verbal feedback. Comment [MS8]: How do you define what an error is? Taking too much time on something? Having to use the back button? Clicking the wrong link in the wrong order? Hovering over a button too long before clicking it? Etc, etc All participants were asked to complete two primary tasks. The two main tasks (1) Find course information and (2) Update course listings each contained two subtasks. Each of the four subtasks were tracked as individual tasks in the Morae application. Splitting the tasks in the application allowed for a higher level of precision in the data analysis.
Throughout the study, participants were encouraged to verbally share feedback and think aloud while performing tasks. This feedback was recorded by hand and provided as anecdotal evidence alongside the findings from Morae. Comment [MS9]: I wouldn t call this informal. It is part of the research process. Informal would be if you were having a beer with a participant after the test and he/she provided additional information J Results/Findings Overall, participants were remarkably successful at completing tasks. Generally, participants more familiar with BU sites and WordPress completed tasks faster than those that did not (Task 1: 73.49s min / 183.02 max, Task 2: 76.84s min / 179.95s max). Completion rate across both main tasks was 100%. Comment [MS10]: How did you define the distinction between completed with difficultly and completed with ease? In your analysis section, it would have been helpful to address these for clarity. Figure 1. Categorical Evaluation of Task Success Success rate was 91.66% across both main tasks; the participant not familiar with WordPress got off track looking for the meta box in Task 2.1. However, the
participant was able to return to the Edit screen and complete the task successfully without assistance. All three of the other subtasks were marked with a 100% rate of successful completion. Comment [MS11]: So, one error means completed with difficulty? Figure 2. Error Rate, by Task As a quantitative measurement of task quality, Error Rate was tracked. While an accurate portrayal of overall error rate is difficult with a limited-scale study, individual errors contributed strongly to individual task analysis. Two errors occurred across sixteen task runs (four tasks across three participants). The first error, encountered in Task 1.1 ( Find Tuition ), was largely preventable. Between the time when the study was designed and the time the study was performed, BU Abroad published a new course listing named Grenoble French Studies Program (Summer). This led to some confusion, as the instruction was defined as Find the 2014 tuition for the French Studies Program in Grenoble, France. Two of the three
participants identified the correct program on the first attempt. The participant that chose the new course was simply advised that this was not the correct course. After backing up a page, the user was able to successfully complete the task. An error was marked in Morae manager during the review of the study. The second error, encountered in Task 2.1 ( Update Course Location ), can likely be attributed to the participant s inexperience with the WordPress platform. After arriving on the edit post screen and briefly looking around, the participant clicked off to the Locations screen. During analysis after the study, an error was marked at this point. Figure 3. Average Time on Task, by Task The participant continued searching through two more screens before returning back to the edit post screen. After returning to the page, the participant quickly found the metabox to update the taxonomy term. Other than these two errors, participants generally encountered few difficulties with the tasks.
Participants commented that assigning taxonomy terms was very easy and just as I d expect it to work. Slight hesitation with the WordPress admin panel was noted in the participant that was not experienced with the platform. When urged to continue on, the participant was able to successfully complete the task without direct instruction. Already familiar with the WordPress environment, the other two participants required no additional assistance. In this case, anecdotal evidence from participant feedback aligned with the quantitative data obtained using the Morae application. Average times for tasks 1.1 and 2.1 were significantly higher than average task times for tasks 1.2 and 2.2. This is readily explained by a learning factor participants absorbed the interface and its functions during the first subtask(s), to later apply them during the second subtasks. Recommendations The behavior of interfaces tested appears to be generally consistent with expectations of users. New users were quick to pick up the functionality after the first task, and successfully applied that knowledge on the subsequent attempts. The error rate, although low, could be attributed largely to a slight learning curve to WordPress admin panel. Another component of the error rate likely could have been eliminated with a more controlled environment. Comment [MS12]: Do you feel the tasks chosen accurately captured usability of the admin panel? I am an avid Wordpress user, but I still make errors from time to time, especially if I haven t used it in a few months. Improved in-line help may help prevent future issues similar to the one seen in this study. This text could be added to the taxonomies metabox, explaining the
functionality to new users. A balance would have to be struck here to ensure this isn t cluttering for existing users. Otherwise, the main recommendation is for further study. A true evaluation of error rate could come from a larger group of participants. Future studies should include participants experienced and inexperienced with WordPress. Participants in future studies should also be recruited from outside of Boston University, particularly for analysis of the front-end experience. Comment [MS13]: Definitely Reflection of this Exercise I enjoyed running this study; the experience and data gained is directly actionable for my work. Being able to improve software through concrete data is an incredibly powerful tool. I m glad to have been able to learn from the experience. For future studies, I would try to work in as controlled an environment as possible. During testing, I hadn t realized that the department added in a new program. Luckily, the interference with testing wasn t substantial but certainly something from which to learn. In this case, I could have cloned the most recent version of the site to our development environment & run the front-end tests from there. This environment is where the back-end tests were run from. However, it was not updated with the latest version of the from-end theme. I also was glad to be able to work with Morae Manager. It s definitely a powerful tool, and I could absolutely see using it for some larger studies. Pulling out hard data from the videos (through tagging & the like) is fascinating. Truly quantifying our actions!
Section Feedback Score 1. Website Good 10/10 Overview 2. Evaluation Purpose & Use Case Scenarios Including the details of the tasks would have been very helpful here. 9/10 3. Recruitment What was your relationship to the participants, and how 9/10 did you recruit them? 4. Data Collection Would have been helpful to note the types of questions 14/15 you asked participants to elicit verbal feedback. 5. Analysis Helpful to clearly define what your metrics MEAN up front, particularly error rate. Please see my note above 13/15 on that point. 6. Results/Findings See notes 13/15 7. Recommendations I had hoped your usability testing would yield more fruit for recommendations. Perhaps it was the choice of participants? 14/15 8. Reflection Glad you found the assignment worthwhile! 10/10 Total 92/100