Use of a user behavior module to make an Learning Management Centre for Instructional Technology Jeffery Tay t@nus.edu.sg NUS Learning Management System IVLE It is called Integrated Virtual Learning Environment What IVLE can do for you Assessments and Quizzes Assignment/Notes Exchange Communities of Interest Class Announcement with SMS Discussion Forums/Chat Rooms Class Management/Class Roster Distribution Lists Grade Book Lesson Plans (Week-by-Week) Library e-reserves/resources Module FAQs Module Syllabus and Readings Multimedia and Video Support Organizer with Class Timetables Project Mgmt. and Peer Review Webcast and Podcast Lectures Surveys and Polls elearning Forum Asia 2011, NTU Singapore 1
Close integration is a strategic advantage of IVLE. IVLE is integrated to.. Admission Curriculum Planning Integrated Student Information System Library Others Admission Course Bidding Course Particulars Curriculum Administration Student Particulars Timetabling Exam Marks Admin. Class Rosters Class & Tutorial Groupings e-reserves Resources Digital Library Lecture Webcast Lecture Podcast Blogs Communities of Interest Module Website Student Events SMS Service System Architecture Built on low cost commodity hardware Fully redundant system Running on Windows Server 2008 R2, IIS7.5, ASP.NET 4.0 and SQL Server 2008 R2 Code base completely developed in-house Primary Site Active DR Site Load Balancer #1 Load Balancer #1 Multiple Web Servers Multiple Web Servers running Windows running Windows Server 2008 R2, IIS Server 2008 R2, IIS 7.5 and ASP.NET 4 7.5 and ASP.NET 4 SQL Server 2008 R2 Clustered Instance (Active/Passive) SQL Server 2008 R2 Clustered Instance (Active/Passive) ASSUMPTIONS elearning Forum Asia 2011, NTU Singapore 2
User Behavior Assumptions From our experience with students, we made certain assumptions about our students. These assumptions form the basis to decide if certain features are worthwhile to develop The assumptions are: Students prefer to quickly get their information from IVLE and get out as soon as possible. We estimate students would spend about 20 minutes on IVLE Students tend to access IVLE between 10pm to 2am. Having an elearning Week generates substantially more load on the system due to the huge number of students accessing at the same time. Students tend to access IVLE from home or outside of campus rather than from within campus especially after office hours. BUT we have no idea if these assumptions are correct, as there is no data to verify them with RATIONALIZATION Rationalization of There is a need to have some form of data capture to validate our assumptions! Two options are available 1. User Voice: Gather the data from user surveys, chats with users 2. Actual user activity: Gather the actual user activity within the system We built a custom module instead of listening to users voice because It is more practical to gain insight from actual users behavior within IVLE and improve their experience Users who provide strong feedback are generally the minority Additional benefits such as auditing, security investigation and capacity sizing Extremely easy to implement using Microsoft.NET built in modules and strong charting capabilities Lower TCO since its all in house developed and maintained elearning Forum Asia 2011, NTU Singapore 3
MEASUREMENT METRICS User Behavior Measurement Metrics We came up with 7 basic metrics to measure student behavior. The metrics are: Total number of user logins per hour Total number of user activities per hour Total number of active users per hour Average number of activities per user session Average duration per user session Daily tools usage Importantly more data can be mined by cross-referencing the same metrics during different time periods or cross-referencing referencing different metrics in the same time period or by correlating them with the academic calendar Additionally, by taking the metrics of ALL the students using the LMS, we are able to gather more insight into the overall behavior of the students FINDINGS elearning Forum Asia 2011, NTU Singapore 4
Students prefer to quickly get their information from IVLE but they do not get out as soon as possible Previously we thought that students try to spend as short a time within IVLE as possible It is not the case, they will stay logged in for close to an hour each time Interestingly staying logged in longer does not equate to more actions, they can be online and idle as well Figure 1 Average duration per session Figure 2 Average activities per session Sunday is the day when students stay the longest Taking their average duration per session based on day of the week revealed that students logged into IVLE longest on Sunday We attribute this to students wanting to prepare for the coming week to maximize their learning experience. Alternatively it could be that they are catching up on the previous week s work and getting it done before its due for submission Figure 3 Average duration per session Figure 4 Average activities per session User activity drops as the term progresses User activity drops as much as 50% between the months of August to September We attribute this to students having more experience with the system and getting more efficient in getting what they want, resulting in a lower activity rate Figure 5 - Average activities per session Figure 6 - Average activities per session elearning Forum Asia 2011, NTU Singapore 5
No significant usage during elearning week NUS conducts regular elearning weeks, where for that one week, lecturers are allowed to use only online tools to conduct their lessons. One of the main tool used is IVLE, however from the statistics shown, there is no significant usage differences between elearning weeks and normal academic weeks Figure 5 - Average activities per session Figure 6 - Average activities per session Peak access between 8pm till 1am Consolidating the Total Number of User Activities Per Hour metric reveals that the peak access to IVLE is between 8pm till 1am, primarily from home. During school hours, they are also relatively active, with an almost equal distribution between internet and intranet access. Lull period is between 3am till 7am. Figure 7 - Peak access by hour Workbin is the most popular tool, followed by forum Taking the total across the entire period of measurement (approximately 2 years), we find that the most used tool is the workbin, followed by the forum. This is because the workbin tool is the primary means to distribute notes, assignments and projects to students, it is also the tool used by students to submit their assignments and project documents back to the lecturer The forum is the next popular because students use it as to discuss materials and share information between themselves and the lecturer Figure 8 - Consolidated tool usage elearning Forum Asia 2011, NTU Singapore 6
Lesson Plan is consistently the most accessed tool before school starts The Daily Tools Usage metric revealed that just before the academic year starts, the lesson plan will be the most heavily used tool. Lesson Plan contains information about the module s lessons during the academic year Possibly students wish to see the lesson schedule before deciding to take the module Figure 9 - Tools usage on 5th January 2010 Figure 10 - Tools usage on 18th July 2010 Figure 11 - Tools usage on 1st January 2011 ADAPTATIONS Adaptations More ways to access the workbin Considering that the workbin is the most popular tool, we decided to create a desktop application which allows users to upload files directly to the module workbins. Both standalone and integrated to Microsoft Office Word, Excel and Powerpoint elearning Forum Asia 2011, NTU Singapore 7
Adaptations Live Update Of New Announcement And Friends Messages Given that students are staying longer within IVLE, we implemented a live update of new announcements and messages from their friends. Adaptations System Procedures Taking into account the usage patterns, we changed our system management procedures to minimize downtime during high usage periods. These include sundays, after 8 pm on weekdays and just before school starts CONCLUSIONS elearning Forum Asia 2011, NTU Singapore 8
Conclusions Conclusions Web 2.0 is a good to have LMS is a service, they expect it to be always on and never go down More importantly, how to make students more productive within a Learning Management System g{tç~ léâ Visit the Centre for Instructional Technology http://www.cit.nus.edu.sg elearning Forum Asia 2011, NTU Singapore 9