What are some common test misuses?

Transcription

1 Welcome to the CLI Winter Lunch and Learn! At your seat, you will find post-it notes. Please use the notes to answer this question. What are some common test misuses? When you are finished, place your post-its on the boards located around the room. The presentation will begin shortly.

2 The Children s Learning Institute Welcomes You To Our Testing: Friend or Foe? Generously Sponsored by

3 An Introduction to Testing Keith Millner, M.A., Ed.M. Director, Children s Learning Institute Programs Children s Learning Institute

4 Why Test? Testing is a means to an end Testing helps us to effectively serve children, students, and families

5 What questions should we ask about testing children? Whether to test children is not in question Questions we should ask about testing children touch on issues like: When to test Who to test What to test How to test

6 Testing: Friend or Foe? Dr. Janelle Montroy Common Misuses of Assessment Dr. Jason Anthony The Purpose of Assessment Dr. Mike Assel What We Can Learn from Tests

7 Testing: A Broad Introduction Assessment vs. testing CLI testing activities Test design and development Electronic test delivery and administration Test review Test related training and support Testing for research projects Testing in clinics Data analysis and reporting

8 Types & Levels of Information Testing Provides Types of Information Question (Which picture shows a cat?) Skill/Concept (Vocabulary) Subject (Reading) Overall (Reading, Math, Science) Levels of Information Child Small group Class School District State Country

9 Misuses of Assessment Janelle Montroy, Ph.D. Assistant Professor of Pediatrics Children s Learning Institute

10 Misuses of Assessment Wrong test for the current situation Not a good test Just because Too much time Bonus: The whole child

11 Wrong Test Who to test What to test Test purpose

12 Wrong Test: Who Who to test misuse: Using a screener to evaluate preschool program success/ children s knowledge at the end of pre-k The TX-KEA screener Administered during the first few months of kindergarten Must be valid and reliable for their intended purposes and targeted populations

13 Wrong Test: what What to test misuse: A diagnostic test to capture individual level differences and change

14 Wrong Test: what What to test misuse: A diagnostic test to capture individual level differences and change

15 Wrong Test: purpose Test purpose misuse: progress monitoring to evaluate teachers Short test evaluating how well student learning matches our expectations for a specific domain.

16 Not a good test 1) Biased test: The case of the letters M and J 2) Not fully evaluated

17 Just Because Misuse not understanding why

18 Misuse Not applying what we have taken the time to find out Too much time

19 The Whole Child

20 Thanks! Images from: memegenerator.net pearsonclinical.com/childhood/products/ /bayley-scales-ofinfant-and-toddler-development-third-edition-bayley-iii.html Magnifyingaids.com Inchcalculator.com sesamestreetlettersegments.wikia.com commons.wikimedia.org Youtube.com Swordofsurvival.com onhyperion3seciso1250a.jpg.html

21 Types of Assessments and Their Purposes Jason L. Anthony, Ph.D., Ed.S. Professor of Pediatrics Children s Learning Institute

22

23 Benefits of Educational Testing Research shows testing can improve instruction Help administrators identify gaps for T&T Inform educational policy Multilevel system of data-driven decision making that helps our educational system evolve

24 Key to Successful Testing Create match among Purpose Type of test Timeline Level of reporting

25 Questions that Require Educational Testing to Answer Are my students learning the content taught? Does this curriculum work? Did some educational policy make a difference? What are the characteristics of impactful schools? Do any classmates require supplemental instruction? What intensity of instruction is needed? Which classmates should be assessed for LD? Does this child have a learning disability? Is this child falling behind and require additional support? What subskills require remediation? What subskills are assets from which to draw upon?

26 Types of Tests Outcome Measures (Summative) Formative Tests Curriculum Linked Tests Benchmark Tests Progress Monitoring Tests Screening Tests Diagnostic Assessments

27 Why? Evaluation Has child learned content taught? Is child s achievement improving RTI: Are changes in instruction needed? Is child at risk? Classification What? Broad achievement Specific skills General skills General skills Broad achievmnt. Achievmnt., Gen. skills, Cognition, Symptoms When? Pre & Post, Annually After curricular units BOY, MOY, EOY Weekly - monthly BOY or upon request Once How long? Hours min min 5-10 min min Hours

28 Why? Evaluation Has child learned content taught? Is child s achievement improving RTI: Are changes in instruction needed? Is child at risk? Classification What? Broad achievement Specific skills General skills General skills Broad achievmnt. Achievmnt., Gen. skills, Cognition, Symptoms When? Pre & Post, Annually After curricular units BOY, MOY, EOY Weekly - monthly BOY or upon request Once How long? Hours min min 5-10 min min Hours

29 Why? Evaluation Has child learned content taught? Is child s achievement improving RTI: Are changes in instruction needed? Is child at risk? Classification What? Broad achievement Specific skills General skills General skills Broad achievmnt. Achievmnt., Gen. skills, Cognition, Symptoms When? Pre & Post, Annually After curricular units BOY, MOY, EOY Weekly - monthly BOY or upon request Once How long? Hours min min 5-10 min min Hours

30 Why? Evaluation Has child learned content taught? Is child s achievement improving RTI: Are changes in instruction needed? Is child at risk? Classification What? Broad achievement Specific skills General skills General skills Broad achievement Achievmnt., Gen. skills, Cognition, Symptoms When? Pre & Post, Annually After curricular units BOY, MOY, EOY Weekly - monthly BOY or upon request Once How long? Hours min min 5-10 min min Hours

31 Why? Evaluation Has child learned content taught? Is child s achievement improving RTI: Are changes in instruction needed? Is child at risk? Classification What? Broad achievement Specific skills General skills General skills Broad achievement Achievement, Gen. skills, Cognition, Symptoms When? Pre & Post, Annually After curricular units BOY, MOY, EOY Weekly - monthly BOY or upon request Once How long? Hours min min 5-10 min min Hours

32 Assessment Case Examples Michael Assel, Ph.D. Clinical Associate Professor of Pediatrics Children s Learning Institute

33 Case Examples Individual Context Overview Comprehensive The Partial (aka, we find what we are looking for) Tracking Changes Individual (child level change) Classroom Level Final Thoughts

34 Individual Context Comprehensive Assessment- Sometimes called Full Psychological Evaluation, Neuropsychological Evaluation, or Full and Individual Evaluation. Typically, an evaluation is initiated because there is a question to answer. Does the child have a developmental disability (e.g., ID/ASD, ADHD, LD) Or the child is struggling in some area (e.g., behavioral, social, emotional, academic functioning). Why Assess? Multitude of reasons (e.g., parent or teacher concern, child has some type of developmental risk factor, family history). Or simply. Something is not right.

35 What is included in comprehensive assessment for a child? While there is variability, most clinicians will assess multiple domains Cognitive Ability or Intelligence WISC-V, SB-5, WJ-Tests of Cognitive Abilities In some cases assessment of nonverbal problems solving skills is accomplished (e.g., C-TONI, Leiter International Performance Scale-3) Language Measures of expressive and receptive language functioning Preschool Language Scale-5, Clinical Evaluation of Language Fundamentals-5 Measures of Expressive and/or Receptive Vocabulary (e.g., EOWPVT, ROWPVT, PPVT-4)

36 What s Included (Cont.)? Academics WJ-IV, TOPEL, WIAT-III, GORT-5, GSRT, TOWRE, TOWL, CTOPP-2 Motor Gross, Fine, and Visual Motor Attention, Concentration, Memory WRAML2, Conners CPT-3, Conners Kiddie-CPT, Test of Everyday Attention for Children Social and Behavioral Host of quantitative and qualitative measures Adaptive Behavior ABAS-II, Vineland

37 What does the data tell you? Comprehensive evaluations provide the clinician with a broad based picture of what is going in with a child. I can pinpoint a learning disability, attention problems, ID, ASD, and some emotional difficulties.

38 Limitations of a Comprehensive Expensive Evaluation Time Consuming Different stakeholders want different information

39 The Partial (Case) Vladimir: Adopted from the Ukraine at age 4. Limited developmental history but we know he was born preterm (e.g., 30 weeks), respiratory difficulties (anoxia, apnea), slow motor development. After a year in the US, parents and pediatrician are impressed with Vladimir s English language skills and he is enrolled in a regular preschool. In Kindergarten, teacher begins to complain about attention problems. Parents take Vladimir to a large group practice in Houston that specializes in treating attention problems. Vladimir completes a number of measures on my list (e.g., IQ and attention) and is ultimately prescribed medication to address attention problems.

40 But (there is always a but) After medication starts the kindergarten teacher is less worried. The family thinks they dodged a bullet. The family moves to the mid east for several years but ultimately ask for a transfer back to the US because Vladimir can t keep up in the International School during the 5 th grade. Vladimir is now struggling in some higher level academic areas (e.g., passage comprehension, written expression). Well intentioned mom tracks me down and hounds me via and sends me a 5 page letter detailing the educational history and lets me review the tests results from the ADHD evaluation.

41 But I know the group who did the workup and I know they make a ton of money by treating attention problems. I also know they don t complete a comprehensive evaluation. Given the history of adoption (international, former Soviet Union, etc.), what was missing?

42 LANGUAGE Vladimir was 9 years old and never had a language assessment. Prior to my comprehensive evaluation, I sent the family to a SLP who tested him with the CELF-5 and found scores that were mildly delayed receptively and significantly delayed expressively. The linguistic weaknesses map onto the areas of school where he was struggling. Weak receptive language skills correlated with his struggles in passage comprehension. Weak expressive language skills made it difficult for Vladimir to express himself orally.

43 Take Away How many of you start an important journey (e.g., a journey that has long lasting implications for you well being) WITHOUT having a roadmap. A quality assessment acts like a roadmap for parents.

44 Tracking Change (i.e., multiple roadmaps) Comprehensive Evaluations produce a great deal of data Data is sometimes hard to interpret I have seen a number of children multiple times for follow-up evaluations (toddler through adolescence). One of my clients with ASD was seen multiple times (22 m, 36 m, 50 m, 66 m, 76 m, 91 m, 120 m, 136 m)

45 Let s talk about the Dad He was thrifty (some would say cheap) He was an engineer and appreciated data His child was diagnosed with an ASD and he had become accustomed to evaluating data in the child s behavior intervention program Like many parents of children with ASD, he carried a binder In the binder he documented progress. Why???? Along with others, I taught the Dad that early intervention was critical and ensuring progress was occurring was critical. He did not want to waste time.

46 Dad evaluated linguistic progress month AE 24 month AE 36 month AE 48 month AE 60 month AE 72 month AE 73 month AE 0 12 mo chron 24 mo chron 36 mo chron 48 mo chron 60 mo chron 72 mo chron

47 Take Away In this case evaluation data was used to track progress. Careful evaluation of progress allowed the family to make a change in providers which led to an increase in the language gains made by the child. Some might say the number of assessments completed was excessive but similar changes to the treatment plan occurred as the child matured (e.g., sent to OT due to weak handwriting at 8 years of age, sent to a psychiatrist due to increased obsessive/repetitive behavior at age 11).

48 Tracking Individual Progress in Classrooms Progress Monitoring- Progress monitoring is used to assess students' academic performance, to quantify a student rate of improvement or responsiveness to instruction, and to evaluate the effectiveness of instruction. Progress monitoring can be implemented with individual students or an entire class.

49 Case Example Tracking Progress in Math Circle Assessment: Pre-K progress monitoring assessment that covers multiple domains. Vocabulary Letter Knowledge Phonological Awareness Math Science Social Studies Motor Skills Social and Emotional Emergent Writing

50 Progress Monitoring Class View

51 Focus on Math 26 item screener that evaluates number recognition, counting, shape naming, shape discrimination, and operations.

52 Hypothetical Counting Objects Data (BOY) Count 3 Count 5 Count 7 Count 10 Count 15 Jason Maria Mike Keith Bella Janelle

53

54 Take Away PM and screening data was used to guide instruction Teacher selects activities that have a chance of helping children improve in a skill area. Progress monitoring typically occurs multiple times across the school year. Groups tend to have variable members (risk level changes across the school year). PM data not used to rank teachers or programs.

55 Improving School Outcomes The Director of a pre-k program recognizes that children coming from her program score poorly on measures evaluating math skills. She meets with the Instructional Specialist who recommends a closer inspection of the types of math instruction that occur within classrooms. They turn to the Classroom Observation Tool. This is an observational assessment that was designed at the CLI. I will give you a flavor of the Math Subscale.

56 Math COT Used by mentors, coaches, instructional support staff, administrators, and researchers. Can be used in a structured manner (e.g., I want to see a typical math activity) or unstructured (e.g., going to look at the classroom and rate all math activities that occur naturally). Somewhat easier for educators because it just has the rater indicate if the behavior was observed (i.e., quality is not rated).

57 Example of the Math Section of the Classroom Observational Tool (COT) Ob Teacher 1 NS Ob Teacher 2 NS Number Recognition. Counting in sequence. Relation between quantities and numbers/counting objects. Ordinal terms. Sorting objects. Graphic representation. Patterns. Shapes. Positional words. Length. Weight. Amount of space/volume.

58 Classroom Observation Tool Can support mentors as they work to help teachers improve. Allows administrators the ability to target PD dollars Allows teachers to see what types of behaviors are expected Most importantly, provides someone working in the role as a coach with information that can guide their mentoring.

59 Assessment Take Home Message Well designed assessments have the potential to help parents, children, students, teachers, mentors, and administrators. Like many things in life, assessments can be used appropriate or inappropriately. There is a chain of responsibility in the use of the assessment. Those collecting the data have to make sure that they are following the procedures inherent in the assessment. Those interpreting the data have the responsibility to use the results in the manner that the measure was designed.

60 Question and Answer Session

61 This event was generously sponsored by Houghton Mifflin Harcourt. Thank you for supporting the Children s Learning Institute.

62 Thank you for coming! A video of this presentation will be made available on our website soon. Please make sure and throw away your trash.