Tips for Creating and Delivering Effective Mobile Assessments Ivan Forward, Questionmark Antonio Herrera, Inlea Copyright 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark
Percentage of World s population covered by a mobile signal Slide 2
Smartphone usage continues to grow Slide 3
2011 Census of South Africa Mobile Phone/Device Usage: Decade of Staggering Growth Rate 84.70% 90% 50% 57% 58.20% 32% 2001 2011 2001 2011 8.40% 11.80% 2001 2011 2001 2011 % of homes with flush toilets % of homes with electricity % of Higher Education completers % mobile phone ownership Slide 4
% of overall website traffic from mobile devices Slide 5
Browser Support more important than ever October 2008 October 2012 Source: StatCounter.com Slide 6
What we will cover Example applications of mobile assessment Tips from our experience Questionmark support of mobile assessment A real-life example
Mobile delivery enables new possibilities Slide 8
Inexpensive, very portable Internet-connected mobile devices
What are the potential applications? Observational assessments Formative assessment Surveys & Course Evaluation
Observational Assessments Slide 11
Traditional vs. Observational Assessment Traditional assessment Participant is presented with questions for direct response Observational assessment Observer is presented with questions to rate participant to measure behaviors, skills and abilities Example Applications Certification Process Observational assessments can be used as critical components of the certification process in some cases are required by regulatory authorities. Medical and Dental School exams: Medical and dental schools make extensive use of observational assessments such as the Objective Structured Clinical Examination (OSCE). Equipment operation: Observational assessment can be used to record and rate operators' skills, abilities, safety practices and adherence to required procedures
Typical workflow for assessor Assessor logs in Selects assessment Selects participant to be observed Answers/submits responses to questions Results/feedback reported to stakeholders Slide 13
Slide 14
Slide 15
Back Slide 16
Formative Assessment Slide 17
Quizzes on mobile devices Low stakes and openly available Often appropriate for mobile devices Enables retrieval practice to reduce forgetting Provides the participant with corrective feedback for revision and learning Encourages self-directed and independent learners
Surveys and course evaluations Slide 19
QR Codes for Launching Assessments Course evaluation example: Embed demographic data in the URL to be saved to special fields in the assessment results, e.g. Course name Instructor name Location If you are delivering a survey (or course evaluation), QR Codes can help by: Increasing response rates because Quick and easy to access the right URL (no typing required) Increasing data captured allows Data easily captured (no typing required) Easier analysis of survey results by demographic is a registered trademark of Questionmark Computing Limited. Slide All 20other trademarks are acknowledged.
QR codes enable quick launching of quizzes and course evaluations Books PowerPoint Signage Slide 21 Benefits Ease of access for increased response rates Demographic data easily captured Better data accuracy Easier analysis of survey results by demographic
Apps to Scan QR Codes iphone Redlaser Neoreader ScanLife Android Google Goggles QR Droid Scanlife Windows Mobile QR Reader BeeTag Quickmark Blackberry ScanLife BeeTag QR Code Scanner www.url.com/xxx is a registered trademark of Questionmark Computing Limited. Slide All 22other trademarks are acknowledged.
The Big Switch Survey http://goo.gl/lrlcy
Mobile Test Centres Use notebooks/3g laptops/ipads to create mobile test centres Enables greater flexibility in where you deliver assessments Assessments can be conducted in on location environments is a registered trademark of Questionmark Computing Limited. Slide All 24other trademarks are acknowledged.
Securing an ipad An app should prevent candidates from easily getting to other URLs But access to home button allows task-switching Hardware solutions are available to enable use of ipad in a kiosk mode is a registered trademark of Questionmark Computing Limited. Slide All 25other trademarks are acknowledged.
Secure Browser One Setting Multiple Platforms Prevents. Printing Screen capture Task switching Control/system keys Right-click / context menu options View source Accidentally exiting exam Questionmark Secure for Windows Questionmark Secure for Mac Questionmark Secure for ipad Slide 26
Kiosk Mode Apps Padlock App Mobile Kiosk App Kiosk Pro App Lockdown to a website Allow linked websites Custom message if linked website is not allowed Back to home timer Browsing limit timer Custom message when browsing time limit is reached Show address bar Show forward / back page navigation arrows Digital signage mode (non touch) Scale pages to fit Clear local HTML presentation files
Tablet Enclosures Slide 28
Considerations: The higher the stakes, the higher the propensity to cheat still require human monitoring! Bandwidth Be sure to test the wifi and/or 3G signal strength prior to event is a registered trademark of Questionmark Computing Limited. Slide All 29other trademarks are acknowledged.
Tips for Creating and Delivering Effective Mobile Assessments Surveys and course evaluations: Sometimes less is more Think small: Screens, input devices and bandwidth To Flash, or not to Flash? Be aware of what technologies your participants devices support Use feedback carefully and make it short enough to read Test and test again: Try before you deliver! Slide 30
Mode of Deployment Mobile (phone) short 1-5 minute modules o Updates on new information o Review of previous curriculum o Learning Check to enhance retention Video (ipad) 10-20 minute modules o Content involves more explanation o Larger volume of content required to present elearning (computer) 15-45 minute modules o Difficult or complex content o Requires interaction and instruction o Higher level of security needed Classroom 1hr+
True for most surveys; Especially true for mobile surveys Consider your objectives: Keep your surveys brief to help increase response rate Put your most important questions at the start of the survey Include only one concept per question Group related questions together Avoid relying on open-ended questions (e.g. essay ) Slide 32
Think small: Screens, input devices and bandwidth
Develop for the Device Browser screen 480 pixels Graphics 200 pixels wide Video H.264 Audio mp3 at 22 khz
Lay-Out Most important goes high and left (4-5 inch square) Finger size matters (44 x 44) Place videos last Avoid Flash Beware of the Fold Line Font size 15-22 1 or 2 columns of content/page
Screen Furniture Ensure your form buttons and other navigation controls (a/k/a screen furniture ) work with low screen resolutions Slide 36
Images Optimise size and compression of larger image files to minimise bandwidth requirements Detail on images should be kept to a minimum to ensure readability on small screens
Be cautious in use of Flash Be aware of what technologies your participants devices support Slide 38
Be cautious in use of Flash Helps minimise bandwidth requirements Strengthens cross-device compatibility Flash will not work on Apple iphones, ipads or ipod touch devices Flash is compatible with some mobile platforms such as Android phones (and tablets) and Windows Mobile Slide 39
When is feedback useful? Correct faulty recollection Helps participant with faulty recall Correcting misconceptions Provide guidance when evidence of fundamental misunderstanding of a topic Feedback only useful if it is read keep short Slide 40
Try before you deliver! Slide 41
Test your content! Test your assessment content With different browsers On as many different devices as practical Emulators can help understand how content will appear on the small screen -- but don t always give a true user experience Setting up a mobile test centre? Test delivery environments when possible particularly for signal/wifi strength Many free online bandwidth tests are available
Questionmark s support for Mobile Delivery Slide 43
Levels of Monitoring Blended Delivery Diagnostic Tests Low Stakes Tests Medium Stakes Exams High Stakes Exams Formative Quizzes Course Evaluations Stakes of Assessment
Access to your Assessments Mobile delivery makes it easier for participants to access assessments: Use mobile Apps Place links in content (PDFs, HTML Web Pages, etc) Embed within a Portal For employees, partners and customers
Android Questionmark Mobile Apps iphone, ipad and ipod touch
Participant Devices Responsive Design 4. On-the-fly auto-sizing of content and controls 1. Senses device/browser 3. Sends assessment to browser 2. Device drivers used to format assessment for device/browser Slide 47
Responsive Design Will resize the assessment tools/content for optimum fit Consider if your assessment content will be affective on smaller screens
Benefits Author once Deliver to any device or context Schedule once Single results set Slide 49
Case study Antonio Herrera, Inlea
Mobile Delivery Real Case Test of Entrepreneurial Profile Belongs to Inlea s CSR activity to help Startup Entrepreneurs to design and launch their business projects (Fundación Inlea, linktostart)
Mobile Delivery Real Case Context, about Inlea: Inlea is an international group of companies dedicated to transfer knowledge between Industry and the Education Sector. We support education programs for companies such as Cisco, Oracle, Corning, and Televes Inlea is Questionmark s distributor for Spain
Mobile Delivery Real Case entrepreneur, i.e. one who "undertakes an enterprise". The term puts emphasis on the risk and effort of individuals who own and manage a business, and on the innovations that result from their pursuit of economic success
Mobile Delivery Real Case
Mobile Delivery Real Case
Mobile Delivery Real Case Why Mobile? This Self-Test will be delivered to large communities of Startup Entrepreneurs, intra-entrepreneurs, and University students. Many of them will be able to take the test in mobile devices and will prefer such option. Mobile delivery will allow a wider deployment to service more users with their device of choice.
Mobile Delivery Real Case Issues? Is a mobile device the best tool for an entrepreneurial profile self-test? What size of mobile device is sufficient? What happens on a crowded context? Is it possible to brand and customize the mobile template?
Screen shot: Laptop Mobile Delivery Real Case
Screen shot: Samsung Galaxy S4 Mobile Delivery Real Case
Thank you for attending! Ivan Forward Antonio Herrera Copyright 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark