SU IRB - Application - Initial of 9 10/3/2014 10:41 AM \APPLICATION FOR INITIAL REVIEW APPROVAL OF A PROJECT INVOLVING HUMAN SUBJECTS Biomedical, Health Sciences Institutional Review Board (BIRB) Social Science, Behavioral, Education Institutional Review Board (SIRB) 207 Olds Hall, Michigan State University East Lansing, MI 48824-1047 Phone: (517) 355-2180 Fax: (517) 432-4503 E-mail: irb@msu.edu Office Hours: M-F (8:00 A.M.-5:00 P.M.) 1a. Responsible Project Investigator: Name: Mark Urban-Lurain ID#: XXX-XX-6348 Department: COLLEGE OF ENGINEERING College: ENGINEERING Academic Rank: Associate Professor Mailing Address: 1410 B Engineering IRB#: x10-577 ID# i036148 Phone: 432-2108 Fax: Email: urban@msu.edu 1b. Secondary Investigator: Name: John Merrill ID#: XXX-XX-6914 Department: BIOLOGICAL SCIENCE College: NAT SCIENCE Academic Rank: Associate Professor Mailing Address: 204 North Kedzie Biological Sciences Program Phone: 517-432-1316 x143 Fax: Email: merrill3@msu.edu 1c. Additional Investigators: Kevin Haudek Joyce Parker
SU IRB - Application - Initial of 9 10/3/2014 10:41 AM Michele Weston Alan Munn Melanie Cooper Sonia Underwood Leah Corley Nicole Becker Jonathan Markey Carl Lira Alexandria Mazur Oscar Judd Keenan Noyes Christopher Minter Kathryn Paris Mihwa Park Rosa Moscarella 1d. Other Personnel: 1e. Study Coordinator: Name: Mark Urban-Lurain ID#: XXX-XX-6348 Department: COLLEGE OF ENGINEERING College: ENGINEERING Academic Rank: Associate Professor Mailing Address: 1410 B Engineering Phone: 432-2108 Fax: Email: urban@msu.edu 2. Title of Project: Collaborative Research: Automated Analysis of Constructed Response Concept Inventories to Reveal Student Thinking: Forging a National Network for Innovative Assessment Methods(C&G 111629) 3. Have you ever received preliminary approval or a 45 CFR 46.118 designation for this project? 4a. Please describe why your project is minimal risk. For example, "My research includes an anonymous survey about...explain what your survey is about" or "my subjects are identifiable, but the questions are not in any way harmful."
3 of 9 10/3/2014 10:41 AM We are investigating students conceptual understanding in a number of undergraduate STEM courses. As part of the research, we will evaluate students' written responses to normal assignments (usually homework, sometimes exams) that they complete as part of their normal instructional activities. Students generally complete these assignments on line, so their answers can be downloaded and de-identified by the Center for Research on College Science Teaching and Learning at MSU. Student PIDs are encrypted by computer hashing algorithms (similar to those used for computer passwords) before being stored. The hashed identifers cannot be reverse decoded by any known computer algorithms. We will collect student demographic data from the registrar's office to investigate any differential student outcomes by gender, ethnicity, GPA and ACT scores. These data are collected routinely by the registrar at admission and throughout the students' academic career. 4b. Indicate Exempt sub-category(ies). TE: Appendix 1 (exempt categories) must be submitted with the Exempt Application. An application cannot be reviewed without Appendix 1. 45 CFR 46.101(b)(1) 45 CFR 46.101(b)(2) 45 CFR 46.101(b)(3) 45 CFR 46.101(b)(4) 45 CFR 46.101(b)(5) 45 CFR 46.101(b)(6) Demonstration Project Category 7 5. Is this project being conducted to fulfill the requirements of an education/training program? 6a. Funding: Project Is Not Primarily An Education/Training Activity (1) Select appropriate funding source(s). Multiple funding sources may be selected. If a funding source is selected, list the name(s) of the funding source(s) and the CGA number. Funding Source Type If funding source is selected, list the name(s) of the funding source OSP/CGA Number U.S. Federal Government (e.g. Department, Agency) National Science Foundation 111629 U.S. State Government (e.g. Department, Agency) Foreign Government Industry Sponsored Foundation or Non-Profit Internal Funds (e.g. MSU department)
4 of 9 10/3/2014 10:41 AM (2) Are any of the funding sources pending? (i) Describe pending funding source(s): National Science Foundation. Funding is pending IRB approval, no CGA number yet (ii) If the project is not funded will you do the research? 6b. The protection of human subjects often requires resources be dedicated for things such as the consent process (space, personnel), the performance of the research (trained personnel interacting with subjects, time, access to subjects, access to facilities) care of subject issues or injuries (counseling, medical care), confidentiality of data (space, equipment) and other monetary and non-monetary resources. Describe the resources that are available for this project for the protection of human subjects. Researchers are either currently working on STEM research projects or new student researchers will be mentored in the RCR guidelines and IRB protocols. All data are stored on MSU's AFS file system or databases in password protected areas. Only researchers on this project will have access to the data. We will follow our established protocols with the assistance of the Center for Research on College Science Teaching and Learning. 7a. List all sites where this research will be conducted. Michigan State University The Ohio State University University of Colorado, Boulder 7b. Do any of these sites have their own IRB? 7c. Have you or will you submit this to any non-msu IRBs? 8a. Describe the purpose, hypotheses and objectives of the research project. Suitable assessments (tests) of conceptual learning (often referred to as concept inventories or diagnostic question clusters ), however, are few and are constrained by the ability to score the outcomes of the tests in a cost-effective manner. Multiple-choice assessments ( selected responses ) are more widespread in higher education, especially at medium to large institutions where class sizes are large, and where automated scoring provides the essential cost-effectiveness. Conversely, written response assessments ( constructed responses ), which are widely held to be superior at revealing actual student thinking, are quite rare in practice given the time and effort required for manual scoring. This project is leveraging the latest computerized tools and statistical techniques to make constructed response assessments more broadly available. Computerautomation will allow the use of these more insightful conceptual questions and tests with much larger numbers of students, thereby providing an enhanced understanding of students conceptual learning. The researchers are working with developers of conceptual testing instruments to create constructed response versions of the tests coupled with the necessary computerized scoring tools with the eventual goal of providing computerautomated evaluation of conceptual thinking. 8b. Describe all procedures, measures and analyses you will use in collecting data from human subjects. This pertains to both prospective and retrospective (i.e. pre-existing) research procedures. All data are collected as a normal part of classroom instruction. Students will complete short constructed response (written) assignments in which they are asked to write about a variety of scientific processes (cellular respiration, carbon cycling, genetics, and evolution.) We will collect students' responses to these open-ended questions given as homework or occasionally as exams. These will be graded by the instructors as usual.
5 of 9 10/3/2014 10:41 AM We use lexical analysis software to extract various terms and concepts from the students' responses and use those data to predict how the human scorers (instructors, TA) evaluate the students' answers. The data are collected with student identifiers so that we can match the students' writing to the human rating of the writing. We will be studying the ability of the automated system to match the human grader. The ultimate purpose of the automated system is to provide timely feedback to instructors on students' performance on formative assessment. Grading will remain in the hands of the instructor. We will also collect student demographic data from the registrar to determine if there are co-variates that influence the analysis and prediction accuracy to determine if there are any biases in these computerized procedures (for example, language use issues that may have cultural origins.) All of these data are collected by MSU as part of the normal process of administration and instruction. Collaborating researchers at The Ohio State University and University of Colorado, Boulder, will be collecting similar data from students on their campuses under their own IRB protocols. We will be collaborating on the development of the computerized lexical resources used to analyze the students' writing, but each site will analyze their own students' data. 8c. Are any procedures done for non-research purposes? (1) Please describe the procedures that will be done for non-research purposes. Assignment and evaluation of the writing as part of the normal course of instruction. 8d. Summarize the project in one paragraph in completely lay terms. Numerous reports on the effectiveness of U.S. higher education in the Science, Technology, Engineering and Mathematics (STEM) disciplines call for increased emphasis on conceptual learning, rather than rote memorization. Suitable assessments (tests) of conceptual learning (often referred to as concept inventories or diagnostic question clusters ), however, are few and are constrained by the ability to score the outcomes of the tests in a cost-effective manner. Multiple-choice assessments ( selected responses ) are more widespread in higher education, especially at medium to large institutions where class sizes are large, and where automated scoring provides the essential cost-effectiveness. Conversely, written response assessments ( constructed responses ), which are widely held to be superior at revealing actual student thinking, are quite rare in practice given the time and effort required for manual scoring. This project is leveraging the latest computerized tools and statistical techniques to make constructed response assessments more broadly available. Computer-automation will allow the use of these more insightful conceptual questions and tests with much larger numbers of students, thereby providing an enhanced understanding of students conceptual learning. The researchers are working with developers of conceptual testing instruments to create constructed response versions of the tests coupled with the necessary computerized scoring tools with the eventual goal of providing computer-automated evaluation of conceptual thinking. The project is a collaboration among three major public universities: Michigan State University, The Ohio State University, and University of Colorado at Boulder. 8e. Are you obtaining consent (telling subjects ahead of time that they are in a research study)? (1) Explain why (why is it not practicable to obtain consent). We would like to collect data from the registrar's office concerning students' demographics and student course grades, along with data about student performance on normally assigned coursework in these courses. This material will not be reported to the public in any way that would reveal any individual student's identity, nor will individual student's identity be known to anyone other than the research staff. We would like to collect these data from all students, with a waiver of consent, as previous research on this
6 of 9 10/3/2014 10:41 AM student population shows that students who are most at academic risk -- one target population for this research -- provide consent to use these data at a significantly lower rate than other students. See the documentation we provided for IRB 05-150. These results, for large-enrollment STEM courses, based on which students often decide whether or not to continue pursing STEM majors, show that requiring consent for research on classroom work will not produce representative results and students who are most at risk will be under-represented in the analysis. This reduces the likelihood that faculty will be able to address the needs of at-risk students based on this research. 9a. Describe your subject population (e.g., high school athletes, small business owners, children with ADHD). MSU undergraduate students enrolled in a variety of undergraduate science courses (BS 110, BS 111, ISB 202, ISP 203 and other undergraduate courses as identified by the research faculty.) 9b. Age range of subjects 18 to 70 9c. The study populations includes: Purposeful Inclusion Children Women of Childbearing Age College Students Minorities Psychiatric patients Wards of State Pregnant Women Institutionalized Persons Low Income Persons Prisoners Persons with diminished capacity None of These 9d. Total expected number of subjects (including controls) for the entire project period 8000-10000 9e(1). will the subjects be identified and recruited? Include who will make initial contact with the subjects. No explicit recruiting of students as this is a normal part of their coursework. 9e(2). Will subjects be recruited using a student research pool? 9f. Will subjects be compensated? 9g. Will the subjects incur additional financial costs as a result of their participation in this study? 9h. Are you associated with the subjects (e.g., your students, employees, colleagues, patients)? (1) Please explain the nature of the association and what measures you are taking to protect subjects rights, including safeguards against any coercion. Some of the researchers teach some of the classes in which we will collect
7 of 9 10/3/2014 10:41 AM data. Since the data are normal classroom assignments, the instructors will be seeing them as part of their normal grading. Because we are requesting a waiver of consent (section 8e) there will be no coercion. 9i. Will this research be conducted with subjects in another country? 9j. Will this research be conducted with subjects in the U.S. from an ethnic group of sub-group or other non-mainstream minorities (including non-english speakers)? 10a. Describe and assess any potential risks (physical, psychological, social, legal, economic) and assess the likelihood and seriousness of such risks. There are no risks beyond those associated with being a student in the class as we are only analyzing the work students created for the class. Maintaining confidentiality of the student data is important to protect student privacy. 10b. Describe the procedures for protecting against or minimizing potential risks and an assessment of their likely effectiveness. Only researchers on the project will have access to the student data. Student data will be anonymized by the Center for Research on College Science Teaching and Learning (CRCSTL). Spreadsheets containing student data are processed using an SHA-1 hash function (http://en.wikipedia.org/wiki/sha-1) to perform one-way encryption of the students' PIDs. This allows student data from multiple sources to be merged based on their SDA-1 hash without having to store PID information. Because this is a one-way hash, it is impossible to recover the original PID from the hash value. These hashing functions have never been cracked. The resulting spreadsheets are stored on MSU's password protected AFS system and only accessible to the researchers. 11a. How will subjects privacy be protected? The only identifiers will be student PIDs and they will be encrypted by CRCSTL. No identifier that can be associated with a student will be kept. 11b. Explain how you will ensure the confidentiality and/or anonymity of the raw research data (e.g. completed survey, interview notes, signed consent). Include in your description where the data will be stored (e.g., locked filing cabinet), who will have access to the data, and how long the data will be stored. If this is question is not applicable, please explain. Please note per the universities best practices the responsible project investigator must maintain the data for a minimum of three years after closing the project. The original raw data will be de-identified by CRCSTL as previously noted. Only the de-identified versions will be stored on MSU's password protected AFS space that will only be accessible by members of the research team. We will keep the de-identified data at least 3 years after the project is ended. 11c. Explain how you will ensure the confidentiality and/or anonymity of the electronic research data (e.g. data entered into database, spreadsheet, stored on a computer, data collected via the web). Include in your description where the data will be stored (e.g. password protected computer), who will have access to the data, and how long the data will be stored. If this is question is not applicable, please explain. Include electronic security measures (e.g., password protected files, data encryption, and other protective measures for computer and/or network storage devices such as jump drives and CDs). The original raw data will be de-identified by CRCSTL as previously noted. Only the de-identified versions will be stored on MSU's password protected AFS space that will only be accessible by members of the research team. We will keep the de-identified data at least 3 years after the project is ended. 12. Does this project involve protected health information as defined by HIPAA?
SU IRB - Application - Initial of 9 10/3/2014 10:41 AM 13a. Does any person responsible for the design, conduct, or reporting of findings of this protocol have a Significant Financial Interest (as defined for the MSU Faculty Conflict of Interest Policy) or other opportunity for tangible personal benefit related to the conduct of the research that might compromise, or reasonably appear to compromise, the independence of judgment with which their responsibilities would be completed under this research protocol? A reportable financial interest includes, but is not limited to, a financial interest in the sponsor, product, or service being tested, or in a competitor of the sponsor or product or service being tested. 13b. Has any financial arrangement, including compensation, ownership interest, stock options, or other ownership interest, (e.g., compensation that is: explicitly greater for a favorable result; in the form of an equity interest in the sponsor of a covered study; or in the form of compensation tied to sales of the product, such as a royalty interest) been established whereby the value of compensation or ownership interest to investigators conducting the study could be influenced by the outcome of the study? 13c. Is this a clinical study where the results may be used to support marketing applications for new human drugs and biological products and marketing applications and reclassification petitions for medical devices to the FDA, as required by law? 13d. Have you or will you submit an FDA form 3454 or 3455 (Conflict of Interest)? 14a. When would you prefer to begin this project? 8/16/2010 14b. Estimated end date of project: 8/15/2012 ADDITIONAL DOCUMENTS/ATTACHMENTS 01. 6/4/2010 Script (i036148_6-3-2010_exempt_appendix1.doc) 02. 6/4/2010 Grant Application Materials (i036148_6-3-2010_proposal_urban-lurain_nsf CCLI II_011310.pdf) 03. 6/18/2014 Correspondence (i036148_6-18-14_add investigator correspondence.pdf) 04. 6/19/2014 Correspondence (i036148_6-19-14_add investigator correspondence.pdf) COMMENTS COMMENT AFTER REVIEW / EDIT BY IRB STAFF (Viewable by PI) PI RESPONSE Comment #1 Reviewer # 9:47:03 Dear Dr. Urban-Lurain: I have added the following investigators to this project at your request: Z41372264: Cooper, Melanie M Z41371721: Underwood, Sonia M A49130766: Corley, Leah Z41371965: Becker, Nicole M A25046503: J. Markey I have removed Libarkin and Long from the study. If you have any questions, please feel free to contact me!
9 of 9 10/3/2014 10:41 AM Sally Conley IRB Administrator II 884-0296