Program Evaluation Committee Phone Conference or Meeting 2:00pm

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee 2.10.16 Phone Conference or Meeting 2:00pm Present Absent Karen Ball Terry Keller Shawna McGill Richard Barnes Shawna Kemper Sasha Poole Marsha Bouchard Siri GuruNam Khalsa Geoff Shuster Diane Evans-Prior Judy Liesveld Mary Wright Donnie Hayes Dusten Marie Bryan Jackson Agenda Item Discussion Action Agenda Review Agenda review, no one had anything to add. Introductions to Committee - No action needed Online Survey Formats Shawna requested ideas from the group. Judy Mary, Bryan, and I will be attending a RedCap Training on 2/22. Elizabeth Dickson, GA student entering in data from past surveys, is currently looking into RedCap for her data entry. Mary mentioned some pros and cons of the software tool and divulged that the service is UNM centered and only available to UNM faculty/staff. Judy The goal is to get the surveys online before the next survey rollout, however an investigation of survey tools is of first proirity. - Judy, Mary, Bryan to attend RedCap course. - Shawna, Judy, and Diane to provide updates on survey tools at 2.26.16 Stwd Mtg

Sasha Having something in place with an overview to faculty first would be best as to avoid having faculty and students flipping back and forth with varying survey formats. Judy assured that post RedCap training the process of moving forward wont be too difficult. Updates from Shawna, Judy, and Diane Evans-Prior will take place at the Feb 26 th stwd mtg. Survey Revisions The Committee agreed that the Student Feedback Survey will be revised to a more quantitative survey before previous survey data that has been entered in is analyzed. Siri recommended that everyone look at the surveys and try to identify the themes, change out check boxes, and mainstream the Course Reports. Marsha agreed that some of the surveys are repetitive and tedious. The committee went on to discuss the issues with certain questions, edits to required data, and grading. The consensus is to get the surveys to be more user friendly Mary recommended to find what is useful in fitting with the NMNEC criteria and tease out that which is irrelevant. Judy agreed and followed with using the course objectives to help match NMNEC outcomes. Update for February 26 th Statewide meeting Shawna will deliver an Update for the PEC committee meeting, provide her Course Report revisions. Mary or Judy will deliver update on RedCap. Elizabeth will present on data. - Shawna to take on first part of revisions to Course Report and provide draft at Stwd Mtg. - Bryan to build doodle poll for future meetings Meeting Adjourned at 2:52 PM

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee 3.30.16 Phone Conference or Meeting 4:00pm Present Absent Karen Ball Shawna Kemper Martha Morales Marsha Bouchard Siri GuruNam Khalsa Mary Wright Diane Evans-Prior Judy Liesveld Bryan Jackson Donnie Hayes Dusten Marie Terry Keller Sasha Poole Agenda Item Discussion Action Agenda Martha brought up the Formative/Summative student No action needed Review/Updates clinical evaluations and what each institution is to do with them. A short discussion took place with the end result was that the evaluations are not to be collected or analyzed by NMNEC. They are evaluations for each institution, solely to be used as program analysis tools. Mary emailed the committee her written breakdown of the Online Evaluation process, trials and tribulations. ADN Course Report Shawna recently discovered via a request from Bryan that there is no formal Level 4 ADN Capstone Course Report. In her findings it was conveyed that the SIC committee began the process but opted to leave the course reports Mary- Request to add item to LC agenda

objectives up to each institution. Shawna questioned if outcomes and objectives should be the same? Terry stated that so long as the student meet the NMNEC graduate outcomes the course report should be the same, considering ADN and BSN both prepare to take the same licensing exam. Mary requested taking this topic to the Leadership Committee for further discussion. Online Platform Mary gave a brief synopsis of her investigation with taking the surveys to an online platform. RedCap (a UNM data analysis tool) is great in its ease of use, security, and access for designated parties. To analyze data from this source would require permissions from the IRB office as RedCap audits the surveys. Judy confirmed this process and included that data sharing would require signed agreements from the other NMNEC institutions. Siri asked if the MOU s in place would suffice. No one could attest to that question as of now. Mary disclosed her understanding of Survey Monkey and that the NMNEC office has several surveys already built in house. The use of this platform would cause NMNEC to be the sole distributer of data belonging to participating NMNEC institutions, is this a productive use of time? The committee agreed that with many questions still lingering with either Online platform there should be no rush to push one or the other on to the NMNEC institutions and so the goal of online surveying will be pushed to the Fall of 2016. In the meantime the Committee will continue to investigate a better system, test said system with a pilot survey program which will include a University program and Terry- Report findings with Qualtrex. Mary- Continue with investigation of online platforms.

Course Report Revision also with a Community College program in Summer of 2016. Spring 2016 will proceed with paper surveys as in terms past, Terry Keller of NMSU will look into their in house Qualtrex system and report her findings at the next PEC meeting. Shawna reported the issues felt by faculty members on the redundancy of the questions on the Faculty Course Reports. Of the complaints, user friendliness seemed to be the top item of concern. The committee was interrupted and was forced to hold the topic item until next meeting 4.13.16 Bryan- Carry Item over to next agenda Meeting Adjourned at 2:52 PM

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee 4.13.16 Phone Conference or Meeting 4:00pm Present Absent Karen Ball Shawna Kemper Martha Morales Marsha Bouchard Siri GuruNam Khalsa Mary Wright Diane Evans-Prior Judy Liesveld Bryan Jackson Donnie Hayes Dusten Marie Terry Keller Sasha Poole Agenda Item Discussion Action Agenda No updates No action needed Review/Updates Course Report Revision Conversation of the Course Report Revision continued on from last PEC meeting on 3.30.16 The committee agreed to remove questions 1 and 2 due to redundant content being requested The committee agreed that not every instructor needed to be added to the report as some instructors have very limited interaction with students. Rather, they agreed to have a textbox included in the form to allow for the instructor who is filling out the form be the sole responsible party and to provide a space to spell out their name (some signatures are difficult to read). Question 4 will be kept. Mary & Bryan- will draft course report based on revision suggestions for review at next meeting.

Meeting Adjourned at 5:15 PM Question 5 will include a new option: Other. This is to be included in a drop down option once reports are taken to online platform Question 6 will remove the plus and minus grade options. A = 100-90, B = 89-80, C = 79-70 and so fourth. Question 7 will be merged with question 8. Discussion of whether Level Objective would be better than using Program Objective. Decision to go with including the Program Objective. Question 9 posed many further questions in regards to it s NMNEC goal, value to analysis, and how to simplify. Also its relation to questions 10 and 11. Mary will draft a version that moves question 7 to an addendum. Question 12 Discussion about relevance of this question to NMNEC Program Evaluation as there is no equivalency of tests between HESI, ATI, and Kaplan. If question kept, including asking about the test score as benchmarked against national averages for same test may be more relevant. Will continue discussion at next meeting.

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee 4.27.16 Phone Conference or Meeting 4:00pm Present Absent Shannon Allen Shawna Kemper Belinda Rincones Karen Ball Siri GuruNam Khalsa Marsha Bouchard Judy Liesveld Diane Evans-Prior Dusten Marie Laura Vanyo Mary Wright Bryan Jackson Donnie Hayes Martha Morales Agenda Item Discussion Action Agenda No updates No action needed Review/Updates Course Report Revision Conversation of the Course Report Revision continued on from last PEC meeting on 4.13.16 The committee revisited question 12 of the course report (11 on Mary s course report draft). Siri brought up the fact that every school uses a different test tool for their standardized testing and also administers the tests in different levels. She then asked the committee how this info is used by NMNEC. Terry explained that it shows proficiency from the students based on the exams National Normal score, the key for NMNEC is to analyze this data on an apples to apples basis. Terry Keller Sasha Poole Mary- Finalize Course Report and add to the SIC and/or Leadership Council meeting agenda s. Mary- Look to a see if a Master Plan exists for Prog Eval. Mary- redesign question #12 (#11draft doc) into

The committee agreed that this collection of data is important but how should it be collected brought about another challenging question. Terry mentioned that the committee doesn t know how all of these tests work, exactly. Siri offered to draft an email asking all of the programs a series of questions related to the test in which they use and how the programs measure these results. Mary restated the goal for the question, and reminded that this question (#12) needs to be worded as to meet all faculty needs. Siri suggested gaining input from the SIC or Leadership Council committees. Mary offered to finalize the course report draft and then bring it to both committees for review and input. Terry asked if anyone within the committee had seen or remembered there being a Master Plan in place for the Program Evaluation reports. Mary volunteered to look into this. The committee reached a point where they questioned whether or not #12 had important relevance to NMNEC and if it was more important to individual programs and their evaluation process. Terry suggested that Mary redesign the question into a table format geared toward testing info, results, and class involved. This idea was offered in an attempt to generalize the question and make it universal to all faculty members. Mary and the committee agreed to the suggestion. Mary will submit edits to the committee, finalize report, and request to add it to the agenda of the SIC and Leadership Council meeting s. The committee went back through Mary s drafted course report. They found Mary s edits, based on past meeting input, were good. Sasha felt that #9 was still an ill placed table format and rephrase question #9 Sir- Draft preliminary email and send to Shawna and Mary for review.

Change in Meeting Schedule question. Shawna agreed and suggested that it be reworded or phrased differently. The committee agreed and Mary will make edits based on their suggestions. The Report will be revisited again on 5/4/16 With end of term deadlines, program commencement, and faculty scheduling issues the committee decided to move the date of the upcoming PEC meeting from 5/12/16 to 5/4/16 at 4pm. Bryan- Make adjustment of meeting date change on NMNEC calendar. Email committee members informing of the recent change. Meeting Adjourned at 5:06 PM

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee (PEC) 05.25.2016 Phone Meeting 4:00-5:00 pm NMNEC PEC Committee Members Present Absent Shannon Allen Karen Ball Marsha Bouchard Diane Evans- Prior Donnie Hayes Terry Keller Shawna Kemper/cochair Laura Vanyo Siri GuruNam Khalsa/co-chair Mary Wright/NMNEC Outreach Judy Liesveld/LC liaison Bryan Jackson Dusten Marie Sasha Poole Belinda Rincones Agenda Item Discussion Action Agenda No updates No action needed Review/Updates Review of 5.4.16 No changes or edits per committee No action needed Minutes Survey Monkey Bryan and Mary worked on building the Course Report, Mary- Look into Course Report Review based on the feedback provided by the PEC, in Survey Monkey. Bryan provided an email to the PEC with a link to the survey for review. Laura began with her two comments: having Siri help build the Course Reports

o 1. NMSU main has satellite campuses (NMSU-A and NMSU-G) and that the dropdown menu for selecting a single program does not meet their needs. She stated that their courses are taught between the multiple campuses. Laura requested that the committee remove NMSU-A and NMSU-G and stick with only NMSU. NMSU-C can remain because they are truly an independent campus. The committee agreed to the work around option and Mary will see that the change is addressed. o 2. NMSU does not exactly follow the grading criteria found on the course report. Mary stated that the format on the course report follows the NMNEC guidelines. Laura will look further into this and will report back on the NMSU grading scale. Follow-up on 5/26 NMSU grading scale is as follows: GRADING SCALE Points Letter Grade 100-92 A 91-84 B 83-77 C 76-67 D Below 67 (Based on the NMNEC Curriculum Undergraduate Program) The NMNEC grading policy is as follows: F Mary- Work with Becky on how NMNEC schools will access their own data Mary to oversee that edits are made in the course report related to removing NMSU satellite sites from the drop down list and in only listing letter grades in the grading range section. Mary- Send out data to be analyzed by the PEC for the presentation to be made at the Education Conference by 5.26.16. All members - to review data and identify themes for each question prior to the next PEC meeting.

77% is the pass standard for all nursing classes The following standardized grading scale is recommended: 90-100 = A 80-89 = B 77=79 = C 67-76 = D Anything below 67 = F Schools can develop their own percentage grade distribution for their letter grades (77-100%) Mary and Judy to get the revised course report (including why standardized testing question dropped from course report) on the agenda for a Leadership Council meeting soon. In conclusion the NMSU grading scale is in compliance with the NMNEC grading policy. To facilitate ease of reporting different grading scales in the course report, will need to remove the percentage grade labels and only use the letter grades. There was an inquiry about the course numbering of the course reports. Mary reminded the committee of the difference between the NMNEC universal numbering and the institutional numbering. Bryan followed up by mentioning the NMNEC Course Crosswalk document found on the Curriculum Tab of the NMNEC website. Siri requested the committee make a timeline of completion for all Course Reports. Mary informed the committee that sustainable funding for NMNEC staffing was not going to be able to keep Bryan on as fulltime. He has since taken a position with another department outside of

NMNEC. Siri volunteered to assist the committee with the building of the reports going forward. Mary mentioned the small edits that all 18 reports would need and Bryan reassured the committee that the edits could be easily managed by a novice to Survey Monkey. Mary went on further to disclose the next steps involved to getting the reports to an online format and that included data share agreements and data collection and analysis. REDCap may have more red tape to get through whereas Survey Monkey may possibly be a good venue based on the current membership MOU s already in place. Mary is still investigating this step. Another challenge is how to schools gain access to their own data? Mary will work with Becky on this and report back next PEC meeting. Siri reminded the group of the Standardized Testing question that was removed from the current Survey. Mary stated that the data will exist on another Survey called the Academic Survey and may be redundant to include in the Course Report Survey. After discussion, Committee conclusion is that we need to keep asking about standardized testing somewhere in the NMNEC program evaluation plan and recommend bringing the revised course report along with discussion of why dropping the standardized testing question to NMNEC Leadership Council. Mary asked the group to consider: How frequently do we survey each student? Is it necessary to do this every level, every term, every student? What would the data look like if the survey participants were selected on a Lottery basis? Siri asked what the data says as of today? Mary mentioned the status of the analysis portion, currently in progress, to be

presented at the Albuquerque Nurse Educators Conference. Judy requested the input and support with data analysis from the PEC members. Everyone agreed. Mary will ensure de-identification of surveys and will share with the PEC members by 5.26.16. PEC Master Plan There was not a specific master plan in place, however the mission of the Evaluation team, per Judy, was to design the 3 current forms of surveys, look at NCLEX and Standardized test results, and develop and administer the surveys required by the APIN grant which has been a funding source for NMNEC (academic or demographic survey, and the employer survey). If there is any place to start with this Master Plan it would be to begin here. Shawna- to discuss her Master Plan draft at the next PEC meeting Meeting Update PEC has decided to meet 6.8.16 at 4pm PEC- Focus on Evaluation Analysis Bryan- Schedule meeting Meeting Adjourned at 5:07 PM

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee (PEC) 06.08.2016 Phone Meeting 4:00-5:00 pm NMNEC PEC Committee Members Present Absent Shannon Allen Karen Ball Marsha Bouchard Diane Evans- Prior Donnie Hayes Terry Keller Shawna Kemper/cochair Laura Vanyo Siri GuruNam Khalsa/co-chair Mary Wright/NMNEC Outreach Judy Liesveld/LC liaison Dusten Marie Sasha Poole Belinda Rincones Agenda Item Discussion Action Agenda No updates No action needed Review/Updates Review of 5.25.16 Sasha sated that she was not the one to inquire about the Minutes to reflect this Minutes course numbering of the course reports. change No other changes or edits per committee Mini-Retreat Discussion occurred about the committee possibly meeting before or after the statewide meeting in Silver City (July Mary and Shawna Will create a survey

Program Evaluation Plan 28 th ) to being work on the NMNEC program evaluation plan. The committee discussed the possibility of meeting the day before the statewide meeting on (July 27 th ) or staying after on the 28 th to meet. Siri stated that she would not be available on the 27 th but could stay after the meeting on the 28 th. Mary will not be available. Shawna would like to have either Mary and/or Judy present as work begins on the program evaluation plan and suggested possibly looking at some different dates the first part of August in a centrally located area (Albuquerque) to possibly meet. Shawna suggested a survey be sent out to the committee members asking about their availability for a face to face meeting. The suggestion was made to possibly meet during the NMNEC meeting in Silver City in the afternoon. Historically committee meetings have been held at this time. Mary reviewed the agenda for the meeting on July 28 th and there is no time scheduled for committee meetings since Dr. Stoll from ACEN will be presenting in the afternoon. monkey asking about continued membership on the committee as well as available dates to meet face to face that will be sent out to the evaluation committee

L1 and L4 Student Evaluation Themes Siri noted data in the documents that had not been deidentified. Mary stated that she would make the necessary changes. The committee started the review of the L1 and L4 student evaluation themes L1: What did you like about L1? Themes identified by the committee included: Positive Faculty Impact and Hands on learning L1: What didn t you like about L1? Themes identified by the committee included: Disorganization, Uncertainty and Lack of clinical time L1: As you journeyed through this level, describe how you adapted or changed to grow in your professional role? Themes identified by the committee included: Need for change, Time management/prioritizing, Appreciation for complex thinking L1: List the ways that your clinical experience supported your learning from the classroom Themes identified by the committee included: Linking theory to practice L1: What has worked well for you so far in this program? Themes identified by the committee included: Putting in the time and Learning from/with peers PEC members will send an email to Mary with any information on any data that has not been de-identified The committee will finish L1 evals on the next phone conference and then will begin Level 4

Meeting Update PEC has decided to meet 6.22.16 at 4pm Meeting Adjourned at 5:00 PM

Connecting all New Mexicans to High Quality Healthcare Program Evaluation Committee (PEC) 06.22.2016 Phone Meeting 4:00-5:00 pm NMNEC PEC Committee Members Present Absent Shannon Allen Karen Ball Elizabeth Dickson/NMNEC Graduate Asst Diane Evans-Prior Shawna Kemper/co-chair Siri GuruNam Khalsa/co-chair Terry Keller/LC liaison Judy Liesveld Sharon Marie Sasha Poole Laura Vanyo Mary Wright/NMNEC Outreach Meeting started at 4:07pm Minutes recorder: Siri GuruNam Khalsa Agenda Item Discussion Action Agenda Review/Updates No updates No action needed Review of 06.08.16 Minutes Corrected the minutes on the date for the July NMNEC meeting from July 28 th to July 29 th. Meeting approved with those corrections. Minutes approved with correction. NMNEC Leadership Council Report Leadership Council approved the Faculty Course Report with a couple of tweaks. Gave the OK for the remainder course reports to be updated to the format of the revised Faculty Course Report and transfer the data to

the survey monkey on-line format. Diane Evans Prior offered to help Mary and offered work study students at her college to work with Mary on these course report to continue the process. Changes to the Faculty Course Report by Leadership Council. Mary did not mention what the tweaks were in her communication to Shawna. 1) The range of numbers for the letter grades had been removed secondary to some schools having slightly different number ranges for each of the letter grades. 2) Leadership Council is OK with us collecting the standardized testing data separate from the Faculty Course Survey. 3) Want us to include the standardize Testing information with the overall scheduling of all of the evaluations for the programs. 4) Leadership Council wants us to propose the scheduling of each evaluation plan. (Student Survey evaluations- which semesters). According to communication the goal is to get the faculty course surveys up and running within the next couple of weeks. There were questions about the asterisks that mark some of the questions on the faculty survey questions. Questions 5, 6, 7 do not have the asterisks. Normally asterisks on survey indicate mandatory filling out by the faculty. Talked about needing a key for the learning activities for faculty. This was based on discussion in the May meetings. It was thought that we would wait and see if that is needed after piloting this for the summer. It would need to go to the Leadership Council for approval. Siri volunteered to create a document with the definitions for the learning activities. Will look to see if there is an on-line site to reference in the document. Will

NMNEC PEC Program Evaluation Master Plan Retreat Results of the survey monkey of who wants to continue on the Program Evaluation Committee and when to meet to continue the discussion on the evaluation plan and schedule discussed. The schedule of when to meet as a committee, either at the July meeting or in August. There were more individuals available for August then for the July. There were 5 individuals each for 4 th + 5 th of August and 5 th + 6 th of August. Judy was to be approach as the tie breaker. have it available if there is a future need. Shawna to contact Judy Liesveld re: availability for August meeting. L1 Data Collection Review from 2015 Breadth of Knowledge Surveys Meeting Adjourned at 5:07 pm Finished up the L1- data collection review. Mary sent out the data for L1 and L4. What has worked well for this program- This was where the committee left off from the June 8 meeting. What improvement would you suggest for this program- Students wanted more clinical. Organization (Things Disorganized). List your top 3 things: Elimination Skills, Skills Lab, Simulation, Clinical What is PDART? (This was listed in the top 3). Think it maybe be the five rights. Patient Care Imaging Reference. Medications- Discussed when Medications skills are being introduced. Learned in skills lab but most schools do not have Level 1 students give medications to patients. Review the L4 data for themes at next meeting.