USING SURVEYS IN INTERNAL AUDITS Dr. Hernan Murdock, CIA, CRMA MISTI
Welcome, Introduction, Administrative Matters Welcome! Webinar runs 12-2 PM (ET) LINK TO MATERIALS: this was included in the email you received with your credentials Breaks: 5 minutes each hour Polling: You MUST answer 90% to qualify for NASBA CPEs! Please fill out the survey following the webinar! Questions during Webinar--Use the Chat Function Content: Direct to Dr. Hernan Murdock Access, Audio or Technical: Direct to MISTI Webinar Series Slide 2
Dr. Hernan Murdock, CIA, CRMA Hernan Murdock is Vice President, Audit Division, with MIS Training Institute. Previously he was the Director of Training at a global internal audit firm, where he oversaw the company s training and employee development program. Prior to that, he was a Senior Project Manager leading audit and consulting projects for clients in the manufacturing, transportation, high tech, education, insurance and power generation industries. Dr. Murdock is a Senior Lecturer at Northeastern University where he teaches management, leadership and ethics. He is the author of 10 Key Techniques to Improve Team Productivity and Using Surveys in Internal Audits. He has also written chapters for two books, and articles on internal auditing, whistleblowing programs, fraud, deception and behavioral profiling and has delivered numerous invited talks and conference presentations at internal audit, academic and government functions in the United States, Latin America, Europe and Africa. Slide 3
Table of Contents 1. What are surveys? 2. Types of questionnaires 3. Designing and conducting surveys 4. Preparing and running surveys 5. Analyzing survey results 6. Reporting Results 7. Best practices Slide 4
SECTION I WHAT ARE SURVEYS? Slide 5
What Are Surveys? A common data collection instrument Used to collect information from or about people to describe, compare or explain knowledge, opinions and behaviors Internal auditors need sufficient, reliable, relevant and useful information to achieve the engagement objectives Surveys are useful to assess objective and subjective information Increases the persuasiveness of auditor findings and comments Slide 6
Types of Projects Where Surveys Can be Used Risk Assessment/ERM Entity-Level Controls Corporate Governance Ethics Sarbanes-Oxley Fraud Human Resources Whistleblowing Programs Operational Audits Quality Assurance Reviews IT Reviews Others Slide 7
Stages of Successful Surveys 1. Setting objectives 2. Designing the survey 3. Preparing a reliable and valid questionnaire 4. Conducting the survey 5. Analyzing the data 6. Reporting the results Slide 8
Stages of Successful Surveys 1. Setting Objectives Clear and concise Baseline for resources, timeline and population Affect survey design and types of questions asked Affects data collection and analysis Slide 9
Stages of Successful Surveys Examples of Objectives Assess the control environment Gather feedback on the audit team s performance Determine the degree of agreement with management s operating philosophy and ethics Gather operational information Solicit feedback on the perceptions regarding ethical environment among Grade 10 or lower employees in US operations Slide 10
Stages of Successful Surveys Key Questions What is the objective of the survey? What do I hope to achieve with the survey? Is this the best way to collect the desired information? How will the requested information help me achieve my goals? How much detail do I need? Slide 11
Stages of Successful Surveys Key Questions What decisions do I hope to impact with the results of the survey? Do we want to compare our results to others to do benchmarking or against best practices? How will I use the data I collect? Who else will use the results and conclusions drawn from the data analysis? Slide 12
Stages of Successful Surveys 2. Designing the Survey Align objectives with survey design. Ask: What is the population and what sampling technique should we use? How many people should we poll to feel comfortable acting on their responses? Who will be included and who will be excluded from the survey? How are we going to collect the necessary data? How are we going to follow up with those that are late responding to the questionnaire? Slide 13
Stages of Successful Surveys 2. Designing the Survey Key questions: What resources (e.g., money, time, skills and tools) are available to pursue our objectives? Do we have the support from senior management to do this? What are we going to do with the data after we get it? What can go wrong? What needs to go right? Slide 14
Stages of Successful Surveys 3. Preparing a Reliable and Valid Questionnaire Clarify the terminology: Terms like ethical behavior, effectiveness and job satisfaction are prone to multiple interpretations Know the respondents: Important considerations include reading ability and knowledge of the subject matter Carefully match what knowledge is sought against the amount of time available to find out Slide 15
Stages of Successful Surveys 3. Preparing a Reliable and Valid Questionnaire You should not conduct a larger-than-necessary survey asking a myriad of questions just because you received management s approval to run a survey Standardize the response format by developing an easy-to-use questionnaire or data sheet Slide 16
Stages of Successful Surveys 4. Conducting the Survey Distribute to the target population Make sure the staff involved is knowledgeable about the survey. Make sure they know the objectives of the study, the timeline and milestones and the importance to the organization Focus on moving things along, looking out for unusual and unexpected situations developing and being receptive to informal feedback from participants Slide 17
Stages of Successful Surveys 4. Conducting the Survey Treat the survey with a sense of urgency, giving reasonable but short deadlines for completion Have a high-ranking officer in the organization write the cover letter or announcement Have a process in place to send reminder notices Use blanket reminders if anonymity is more important than response rate Otherwise, you can target your reminders Slide 18
Stages of Successful Surveys 5. Analyzing the Data Enter the data into analytics tool as soon as possible Look for issues, anomalies in the data for immediate attention Don t overuse statistics Slide 19
Stages of Successful Surveys 6. Reporting the Results Be flexible with presentation and format Consider visual presentation over heavy text content Format: Methodology then results or Results then methodology Report as quickly as possible Share results with as broad an audience as practical, possible, reasonable and appropriate Slide 20
SECTION II TYPES OF QUESTIONNAIRES Slide 21
Types of Questionnaires Oral Surveys Individual or group setting Personally or over the telephone More personal than written or electronic surveys Group or audience polling technology Turning Technologies Option Technologies Interactive Elway Research ThinkTank Slide 22
Types of Questionnaires Oral Surveys Advantages Personal contact Quality of the data Body language Response rate Oral Surveys Disadvantages Cost and time Types of questions possible Bias Negative attitude Slide 23
Types of Questionnaires Group Questionnaires Advantages Response Rate Group Questionnaires Disadvantages Sample size Scheduling Slide 24
Types of Questionnaires Drop-Off Surveys Advantages Convenience Response rate Drop-Off Surveys Disadvantages Sampling Time Response rate Slide 25
Types of Questionnaires Mail Surveys Advantages Sample size Convenience Cost Personal bias Mail Surveys Disadvantages Low response rate Limits on respondent abilities Slide 26
Types of Questionnaires Electronic Surveys - Advantages Cost savings Easier data collection and initial analysis Higher response rate More candid responses Faster distribution Potentially faster response time and wider coverage Interactivity Slide 27
Types of Questionnaires Electronic Surveys Disadvantages Layout and formatting Additional instructions Sample limitations Level of confidentiality Potential technical problems Slide 28
Types of Questionnaires Electronic Surveys Providers Zoomerang SurveyMonkey SurveyGizmo QuestionPro SharePoint FreeOnlineSurveys Lotus Notes Others Slide 29
SECTION III DESIGNING AND CONDUCTING SURVEYS Slide 30
Survey Design Correlational Study Example: Relationship between 1. Employee morale and customer satisfaction 2. Trust in management and willingness to use whistleblowing program 3. Damage claims, complaints, lost-time injuries, absenteeism and sick leave Slide 31
Survey Design Time Studies 1. Prospective study: Study data over time into the future. Capture present state, then examine what happens subsequently Cohort study of Accounting grads 2. Retrospective study: Study events that occurred before survey. Study of academic or professional backgrounds of internal auditors Number and severity of findings in departments where manager participated in I.A. Rotation Program Slide 32
Survey Design Time Studies Control Group with Treatment Control (group live training) Experimental (CBT) Self-Control / Longitudinal / Pretest-Posttest: Study members of group two or more times. Take a test, take ethics class, then retake test Slide 33
SECTION IV PREPARING AND RUNNING SURVEYS Slide 34
Designing Questions Closed-ended vs. Open-ended Questions Likert scale Use Open-Ended Questions The respondents own words are essential Respondents are willing and able to provide answers The response alternatives are unknown You have the skills to categorize and analyze comments More time available Use Closed-Ended Questions You want or prefer to rate or rank responses You know what response alternatives are beforehand You prefer to count and tabulate responses You will use charts, graphs and other statistical tools to report results Less time available and have tool to analyze results Slide 35
Closed-Ended Questions Advantages Ease of analysis More specific Faster completion time Disadvantages Limited answers Limited information Discourages disclosure Slide 36
Open-Ended Questions Advantages More flexibility Avoid rote answers Develop trust Disadvantages Time to complete Unnecessary information and additional effort Greater subjectivity Confidentiality Slide 37
Key Considerations Avoid ambiguity Phrase questions carefully Consider level of language proficiency Use focus groups or series of individual interviews Changing from open-ended to close ended Be careful when translating questionnaire Slide 38
Key Considerations - Administration Endorsement by senior management Set reasonable but tight deadline Explain importance if survey is optional Let participants know how long it will take to complete the survey Partner with management about the survey Consider and pre-establish roll-up levels to protect confidentiality and anonymity Keep the questionnaire short, questions short and language simple Slide 39
SECTION V ANALYZING RESULTS Slide 40
Analyzing Results Implications on survey design Software for Statistical Analysis Excel SAS Snap Survey SPSS Minitab Statgraphics Winstat Variables Dependent Independent Slide 41
Processing Results Edit the data to make sure it is correct and complete Descriptive vs. Inferential Statistics Percentages, totals, anticipating, explaining Measures of Central Tendency Mean, Median, Mode Measures of Variability Range, Variance, Standard Deviation Slide 42
Presenting Results Graphs and charts Tables Summary statistics Slide 43
SECTION VI REPORTING RESULTS Slide 44
Report Contents The purpose or objective of the study The population, sample surveyed and criteria for inclusion The methodology used to develop the survey and if a pilot study was conducted The response rate and demographic background of participants The analytical methodology applied to the data A discussion of the results, conclusions and recommendations A copy of the questionnaire Slide 45
Report Contents Reports often show: Each question as it appeared on the questionnaire The list of response alternatives (if applicable) A column for the frequency with which this option appeared (as a percentage) A count showing the number of times that response alternative was selected (as a whole number) Descriptive statistics (mean, median, mode, standard deviation, etc.) applied to the results Slide 46
Report Contents Use tables, charts, graphs, histograms Use colors, shapes and textures to draw attention Cross-reference the information to locations, titles, genders, years of experience, etc. Group Strongly Disagree with Disagree and Agree with Strongly Agree Slide 47
SECTION VII BEST PRACTICES Slide 48
Best Practices Anonymity is often key, so tell participants and follow through. It is particularly important when studying highly personal, sensitive and controversial topics and to obtain honest answers Design the questionnaire carefully, balancing close and open-ended questions Ask clearly for exact information, especially when using open-ended questions Make questionnaire as short as possible Engage a focus group and field test for quality control Limit number of response alternatives Orient response alternatives consistently Slide 49
Best Practices If you need to translate the questionnaire, use native speakers to do so Keep good notes about the process and procedures followed Decide if the survey might be repeated in the future, it would affect the design and procedures followed for subsequent comparisons and trend analysis Keep the questions simple and the response options logical Avoid jargon, abbreviations, very technical, colloquial or slang terms Simplify the language Slide 50
Best Practices Avoid words that carry a connotation that could bias the participants responses Define specific time periods and components Use complete sentences, whether as statements or questions Avoid negative questions such as Did the Auditor in Charge fail to give you a copy of the exit meeting notes? Avoid confusion with Agree/Disagree items, such as Employees should not be required to work excessive overtime or travel on weekends Slide 51
Best Practices Avoid two-edged questions, which have two separate ideas. For example: Would you inform your manager or the ethics hotline if you became aware of an unethical act? Enter data into analytics tool ASAP to check procedures Questions that rely on time periods should not be vague or undefined Train all data collectors Run the survey with the endorsement of senior management Report the results clearly, promptly, respectfully and anonymously Slide 52
Best Practices - Examples Entity Level Controls IT Operational Quality Assurance Customer Satisfaction Representation Sarbanes-Oxley Compliance Risk Assessment Slide 53
Summary Surveys are a flexible tool, increasingly being used by auditors to assess multiple aspects of their organizations Design it carefully, keeping in mind what your objectives are and who the best participants are Use the results to present objectively what many have historically considered subjective topics Use the results to enhance the control environment of your organization Slide 54
Thank You! Please fill out the survey following this webinar! Slide 55