Improving Evaluation of Professional Development in Mathematics and Science Instrumentation and Data Collection Kwang Suk Yoon May 5, 2005 Collect Data on: Quality of PD Teachers content knowledge Quality of classroom instruction Student achievement
Collect data that are: Valid for specific purposes Reliable accurate and unbiased Aligned across measures Time-sensitive meet emerging needs; provide formative information for decision-making or continuous improvement Cost-effective provide useful data given limited resources Approaches to Data Collection Survey Log Interview Focus group Observation Document review
Professional Development Activity Log (PDAL) The PDAL is a web-based, self-administered, longitudinal data collection tool for teachers to record their professional development experiences in detail with the assistance of a series of structured prompts Teachers log on to their password-protected web account and fill out their PDAL at regular intervals Visit www.pdal.net for more information
PDAL Entries Name of activity Number of hours spent on each activity and its duration Whether the activity is a one-time or continuous event (e.g., recurring over a number of months) Type of activity (e.g., workshop, summer institute, study group) Purpose of activity (e.g., strengthening subject matter knowledge) PD quality features (e.g., active learning, coherence, collective participation) Content focus (e.g., algebraic concepts: absolute values, use of variables, etc.) Instructional practice instructional topics covered in each activity (e.g., use of calculators, computers, or other educational technology) Active Learning Observe demonstrations of teaching techniques? Lead group discussions? Develop curricula or lesson plans, which other participants or the activity leader reviewed? Review student work or score assessments? Develop assessments or tasks as part of a formal professional development activity? Practice what you learned and receive feedback as part of a professional development activity Receive coaching or mentoring in the classroom? Give a lecture or presentation to colleagues?
Coherence Designed to support the school-wide improvement plan adopted by your school? Consistent with your department or grade level plan to improve teaching? Consistent with your own goals for your professional development? Based explicitly on what you had learned in earlier professional development activities? Followed up with related activities that built upon what you learned as part of the activity? Why PDAL? Collects disaggregate information about specific PD activities Increases the level of specificity of PD data and reduces bias introduced by gross data aggregation Gathers accurate, reliable, and time-sensitive information Minimizes recall problem with retrospective reports Allows teachers to review their own logs Teachers can reflect on their own PD experiences Tailors technical assistance to teachers based on their response patterns Generates context sensitive questions
Black Box of Survey Data: Hypothetical Data Teacher Activity Time Total Contact Mean Active Jul- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Hours Learning Mr. Anderson A,B,C 32 30 12-21 - 14-109 0.8 Ms. Lopez A,D 32 32 - - 10 10 12 10 106 1.3 Mrs. Kelly E - 20 20 - - - - - 40 0.5 Mr. Lee E,F 18 18 44 18 - - - - 98 0.8 Mrs. Smith G - - - - 8 - - - 8 1.0 Aggregate 0 82 100 76 18 39 10 26 10 361 0.9 Fine-grain, log-level data on contact hours: Disaggregated by teacher by activity by time Hypothetical Data Teacher Activity Time Total Contact Mean Active Mean Contact Jul- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Hours Learning Hours Mr. Anderson A 32 30 - - - - - - 62 2 31.0 B - - 12-13 - 14-39 3 13.0 C - - - - 8 - - - 8 1 8.0 Ms. Lopez A 32 32 - - - - - - 64 2 32.0 D - - - - 10 10 12 10 42 4 10.5 Mrs. Kelly E - 20 20 - - - - - 40 2 20.0 Mr. Lee E 18 18 16 - - - - - 52 3 17.3 F - - 28 18 - - - - 46 2 23.0 Mrs. Smith G - - - - 8 - - - 8 1 8.0 Total Contact Hours 82 100 76 18 39 10 26 10 361 20 18.1 No of Logs 3 4 4 1 4 1 2 1 Mean Contact Hours 27.3 25.0 19.0 18.0 9.8 10.0 13.0 10.0
Unique Benefits of PDAL Provide rich, in-depth descriptive data with a high level of specificity Topical coverage at the fine-grain level Topic intensity (i.e., amount of contact hours per topic) Provide time-dependent measures Average contact hours by month Percent of months with PD Can be used for on-going formative evaluation to continuously improve PD Allow activity-specific (and time-specific) real-time feedback Allows for real-time data analysis Teachers can reflect on their own PD experience Average Contact Hours by Month 40 35 30 25 Hours 20 15 10 5 0 Jul- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Mar- Apr- May- Jun- Jul- Aug- Month
Considerations for Data Collection & Analysis: Timing, frequency, & interval Seasonal variation in PD; time period If data are collected at multiple time points, what is an optimal interval between the time points? Constant interval? Cross-sectional or longitudinal data Linking data over time Multi-level data Linking data across levels Cross-nested data (teachers share some common activities) Identification of unique PD activities Scale of data collection cost-effectiveness