106 Tips for Creating and Delivering Effective Mobile Assessments Jeff Place, Questionmark June 21 23, 2011 San Jose, CA
What will we cover today? What are mobile assessments? Assessments in Learning / Retrieval Practice Making your assessments mobile Example applications of mobile assessment Embedded assessment Tips for mobile assessment
Percentage of World s population covered by a mobile signal Slide 3
Smartphone usage continues to grow 2010 in the US: 63,228,000 smartphone users (source: ComScore) Slide 4
What are mobile assessments? Any assessment taken on a mobile device i.e. Internet enabled mobile phones, notebooks, PDAs, laptops Enables your participants to undertake anytime, anywhere learning
Assessment Thru Learning Process Learning Mentoring Instructor Collaborative & E-learning Work Place Pre-Learning Assessments In-Learning Assessments Of-Learning Assessments Evidence Accumulated
Assessment Thru Learning Process Job Task Analysis Learning Course Evaluations Needs Analysis Mentoring Instructor Collaborative & E-learning Work Place Pre-Learning Assessments In-Learning Assessments Of-Learning Assessments Evidence Accumulated
Pre-Learning Assessments Purpose: Create intrigue Route to course Inform instructors Establish benchmarks Mentoring Learning Instructor Collaborative & E-learning Work Place Pre-Learning Assessments In-Learning Assessments Of-Learning Assessments Evidence Accumulated
In-Learning Assessments Purpose: Learning Encourages focus on learning Provide instant feedback to instructor Stimulate and strengthen memory Correct misconceptions Mentoring Instructor Collaborative & E-learning Work Place Pre-Learning Assessments In-Learning Assessments Of-Learning Assessments Evidence Accumulated
In a Perfect World Slide 10
In the Actual World They don t get it All! Slide 11
Post-Learning Assessments Mentoring Learning Instructor Collaborative & E-learning Purpose: Was learning environment okay? Knowledge transfer occurred? Reduce Forgetting Curve Certifications Work Place Pre-Learning Assessments In-Learning Assessments Of-Learning Assessments Evidence Accumulated
Learning Curve Purpose: Content repetition Memory retrieval practice Strengthens memory recall Ouch! The Forgetting Curve Knowledge/Skills Learning Experience Questions Learning Experience Questions Learning Experience Time Slide 13
In the Actual World They Forget It! Slide 14
Slow the Forgetting Curve Purpose: Memory retrieval practice Strengthens memory recall Reduce Forgetting Curve Knowledge/Skills Learning Experience Questions Learning Experience Questions Learning Experience Questions Questions Time Slide 15
Advantages of Questioning Retrieval Learning Forgetting Frequent questioning reduces forgetting and improves retrieval
Advantages of Questioning Stimulus Provides cue for your search and retrieval process Search Assessments provides search and retrieval practice Action Answering the question/performing the task
Blended Delivery of Assessments Levels of Monitoring Diagnostic Tests Low Stakes Exams Medium Stakes Exams High Stakes Exams Formative Quizzes Course Evaluations Stakes of Assessment
Access to your Assessments Mobile delivery makes it easier for participants to access assessments: Use mobile Apps Place links in content (PDFs, HTML Web Pages, etc) Embed within a Portal For employees, partners and customers
Auto Sense Device/Browser Package the Assessment for Device/Browser/iframe Sends Senses Device/Browser HTML Auto-Size senses pixel size of window Slide 20
Benefits of Auto sensing Author once Deliver to any device or context Schedule once Single results set Slide 21
Mobile devices Inexpensive, very portable Internet-connected mobile devices are often ideal for delivering online observational assessments
Apps for Mobile Assessment Android Apple (iphone, ipad and ipod touch)
Android App Slide 24
Android: Assessment list screen; Launch assessment screen Slide 25
Android: Participant completes assessment; Receives feedback Slide 26
Demonstration: ipad App http://www.youtube.com/watch?v=e4lazgywzdi& feature=player_embedded Slide 27
What are the potential applications? Observational assessments Mobile test center Deliver to the device in your hand
Slide 29
What are Observational Assessments? Observational assessments provide a mechanism for an observer to assess a participant For example, a participant might be observed and rated on Performance of a specific task or procedure Knowledge acquisition and application Demonstration of skills/abilities that traditional assessment may be less reliable in measuring Slide 30
Observational assessment vs. traditional quiz, test or exam Traditional assessment Participant is presented with questions for direct response Observational assessment Observer is presented with questions to rate participant according to pre-determined criteria Slide 31
If assessing performance of a task, observer typically rates participant in three areas: Before the task Performing the task Conclusion/followup on the task
Typical workflow for observer Monitor/observer logs in Selects assessment Selects participant to be observed Answers/submits responses to questions Results/feedback reported to stakeholders Slide 33
Slide 34
The same assessment that is delivered on traditional laptop can be delivered via a handheld mobile device such as ipad, iphone or Android phone/tablet
Feedback to Participant on Performance
Example applications of Observational Assessment Driving school examiner: A driving school examiner observes and rates the license applicant against several pre-elected tasks, such as parking, observation and awareness. The examiner marks down results for each task (awarding scores for each) that are then automatically calculated in to a passing or failing mark. Performance or level 3 assessment: An organization has delivered a training course to its employees and is now evaluating its effectiveness. By observing and monitoring behavioral changes in the workplace before and after the training, the organization will be able to assess the changes caused by the training and evaluate the return on investment gained. Equipment operation: Observational assessment can be used to record and rate operators' skills, abilities, safety practices and adherence to required procedures Customer Service evaluation: Workplace Assessments can also be used to evaluate retail sales and customer service staff on how well they cope in different situations to assess where additional training is required or where personnel excel.
Slide 38
Mobile Test Centers Use notebooks/3g laptops/ipads to create mobile test centers Enables greater flexibility in where you deliver assessments Assessments can be conducted in on location environments Slide 39
Securing an ipad The Questionmark ipad App prevents candidates from easily getting to other URLs But access to home button allows task-switching Hardware solutions are available to enable use of ipad in a kiosk mode Slide 40
Considerations: The higher the stakes, the higher the propensity to cheat still require human monitoring! Bandwidth Be sure to test the wifi and/or 3G signal strength prior to event Slide 41
Slide 42
Surveys and course evaluations Potential for higher response rate by encouraging participants to access via mobile delivery Collect survey results on the move Embed surveys into your learning content for instant feedback
Survey Tips: Keep your surveys brief Put your most important questions at the start of the survey Make use of the participant s comment box available in Perception Include only one concept per question Use the Survey Matrix question to group related questions together Slide 44
Slide 45
Quizzes Low stakes and openly available Often appropriate for mobile devices Enables retrieval practice to reduce forgetting Provides the participant with corrective feedback for revision and learning Encourages self-directed and independent learners
What is feedback? Stimulus is provided to a participant according to their responses within an assessment. Feedback can be provided Based on how a participant scores and/or answers A single question A subject or topic of questions Or, based on how a participant scores on an assessment Slide 47
Simple corrective question feedback Question and participant s answer Corrective feedback given Slide 48
Feedback based topics of questions Topics of questions answered in assessment Score on topic Corrective feedback points participant to appropriate learning resources Slide 49
General guidelines on feedback Improving performance Give the participant guidance for improving their skill or knowledge Precision Be precise enough that the participant knows what needs to be improved Learning resources Tell the participant how and where to go to acquire the information Slide 50
When is feedback useful? Dealing with difficult questions Helps participant understand, learn from the experience Correcting misconceptions Provide guidance when there is evidence of a fundamental misunderstanding of a topic Slide 51
Results of longitudinal study: What percentage of the knowledge that you need to do your job is stored in your own mind? 1986 1997 2006 75% 15-20% 8-10% Robert Kelley, Carnegie Mellon University Slide 53
Our Progress since 1990 Classroom Trainers, Courses, Content, Assessments elearning Online instructional designed contentbased courses, LMS, Assessments Blended Learning Trainers, Online courses, LMS, Coaching, Assessments Social Learning Trainers, Online courses, LMS, Wikis, Blogs, Assessments 70+20+10 Age of Learning Facilitators, Colleagues, SMEgenerated content, Collaboration platforms, Assessment of Knowledge, Skills, Abilities, Behaviors and Performance Slide 54
The 70+20+10 model for learning and development suggests that we learn ~70% from Experience ~20% from Others ~10% from Study Wikipedia: http://en.wikipedia.org/wiki/70/20/10_model Slide 55
Technologies used to support social/informal learning initiatives Portals Social Networking Wikis Forums Blogs Slide 56
Embed in a Wiki Embedded in a Wiki Slide 57
Embedded in a Social Networking Platform
Slide 59
Embedding in Blogs, Portals, Wiki s For Learners For Instructors Focus the learners attention Practice memory recall Identify what was learned Provide feedback to assist the learner to: Correct misconceptions Instill confidence Point to relevant material Easily use assessments throughout the learning process Provide seamless user experience Centralized management of assessment content and results Slide 60
Slide 61
Know the problems you are solving Why do you want to use mobile delivery? Observational Assessments Mobile assessment centres Flexible participant access What are the potential benefits for your participants? Anytime/anywhere learning More authentic assessments Consider mobile variations Screen size Formats Apps
Images Resize buttons and screen furniture to get optimum fit for device However, detail on images should be kept to a minimum to ensure readability Consider the potential screen size use smaller less detailed images When possible, preview your assessments on a target mobile device
Text is unreadable and requires participant to zoom and scroll around to see details in image Text is readable and it is not necessary to zoom or scroll to see image.
Be cautious in use of Flash Flash is supported by many testing tools, but not by Apple Look for an authoring tool that does not rely on Flash for delivery Flash will not work on iphones, ipads or ipods Flash is compatible with some mobile platforms such as Android phones (and tablets), Windows Mobile, Blackberry (later this year) To ensure cross-device compatibility and minimize bandwidth requirements, avoid using Flash Slide 65
Have the technology work for you Author once deliver anywhere Auto-sensing Embedded Mobile Apps Slide 66
Auto-sensing Will resize the assessment tools/content for optimum fit Consider if your assessment content will be affective on smaller screens Slide 67
Mobile Assessments: Item Types Question Type Apple Windows Mobile Blackberry Android Multiple Choice Multiple Response Likert Scale Fill in Blanks Select a Blank Matching Ranking Pull-down List Numeric Text Match Survey Matrix Knowledge Matrix Essay Explanation True/False Yes/No Adobe Captivate Adobe Flash File Upload Hotspot Drag and Drop Slide 68
Resonate with 70+20+10 ~70% from Experience ~20% from Others ~10% from Study Slide 69
Mobile assessments and learning Adds value to anytime/anywhere learning content Enables flexible/personalised learning Encourages social and collaborative learning Expands your ability to complete assessments in the most relevant and appropriate context Enables you to leverage devices already in the hands of your participants Slide 70
Slide 71
Tips for Creating and Delivering Effective Mobile Assessments Jeff Place jeff@questionmark.com