CONSTITUENT VOICE TECHNICAL NOTE 1 INTRODUCING Version 1.1, September 2014

Size: px
Start display at page:

Download "CONSTITUENT VOICE TECHNICAL NOTE 1 INTRODUCING Version 1.1, September 2014"

Transcription

1 preview begis oct 2014 lauches ja 2015 INTRODUCING A serviced cloud platform to share ad compare feedback data ad collaboratively develop feedback ad learig practice CONSTITUENT VOICE TECHNICAL NOTE 1 Versio 1.1, September 2014 At its heart, Keystoe s Costituet Voice methodology promises to shift the very power to defie success, ad to declare whe it has bee achieved, ito the hads of those that developmet orgaizatios claim to serve. This shift toward greater agecy is essetial if we are to make developmet more effective. Raymod C. Offeheiser Presidet, Oxfam America

2 Table of Cotets How to read this techical ote 1 Uderpiigs 2 No shortcuts to progress 4 Eablig coditios ad readiess 6 Elaboratig the Voice Cycle Stage 1 Desig 8 Balacig four measuremet desig priciples 8 Evidece o four dimesios of performace 10 Importace 10 Quality of Service 11 Quality of Relatioship 12 Outcomes 13 Stage 2 Collect 14 Aoymous vs. o-aoymous & idepedetly collected vs. self-collected 15 Bi-modal data collectio: light ad heavy 17 Stage 3 Aalyze 20 Segmet 20 Triagulate 23 Bechmark 25 Visualize 26 Towards idexes 27 Stage 4 Dialogue 28 Broadcastig ad publishig 29 Group discussios ad key iformat iterviews 29 Iformal follow up ivestigatios 30 Co-creatio 31 Public reportig 31 Stage 5 Course Correct 32 Some cocludig observatios 33 Testig your theory of chage or ew proposals 33 As part of your evaluatio framework 34 Commuicatig across costituets 34

3 How to read this techical ote Costituet Voice is a methodology developed by Keystoe Accoutability to eable orgaizatios to improve results by optimizig their relatioships with their costituets. The purpose of this ote is to help orgaizatios uderstad what Cosituet Voice method is ad how it works. Ayoe ca use the Costituet Voice methodology for free. I order to provide you with the emergig learig ad experiece relatig to Costituet Voice ad feedback systems, we ivite you to register your iterest at Costituet Voice is a work i progress. We have a strog focus o rapid cycle learig across all our work, icludig ad maybe eve especially with respect to our core methodology. We are costatly searchig for better questios ad ew ways to ask the, for ew approaches to aalyzig ad iterpretig data, ad for simple ways for orgaizatios to respod ad improve. While comprehesive i the sese of describig all aspects of our methodology, this ote is ot exhaustive i its illustratios. Examples of Costituet Voice i actio are recorded ad published o our website cotiuously, ad we will update this Techical Note regularly. Orgaizatios have a diverse rage of costituets i additio to the people who are the primary focus of the orgaizatio s missio. Some costituets cotribute directly to, ad are directly affected by, the orgaizatio s programs ad activities. These iclude staff, doors, ad parter orgaizatios. Other costituets play a less direct role, like board members, govermetal or traditioal authorities, ad wider commuity structures. Sometimes a orgaizatio s costituets are people. Sometimes they are other orgaizatios. Nevertheless, the orgaizatio usually has a degree of moral or legal obligatio to Govermet liste ad egage with each of its costituet groups. Keystoe helps cliets to cultivate voice with all these costituets. This ote is writte maily about applicatios of CV where respodets are idividuals. To lear more about how we cultivate orgaizatioto-orgaizatio feedback, please visit the Performace Surveys pages o the Keystoe website. There you will lear how some of our cliets are usig CV to improve their relatioships with ivestees, suppliers, gratees, ad parters. Our data set here icludes over 4,000 resposes to Keystoe Performace Surveys from over 30 coutries. Commuity groups Our orgaisatio Board Staff Primary Costituets Peer orgaisatios Doors The relatioship betwee a orgaizatio ad its costituets is govered by the relatioship of each with the primary costituets. Each costituet s actios ca ehace or limit the effectiveess of the others. 1

4 Uderpiigs Costituet Voice is a bled of two importat lies of thought i developmet. Costituet Voice s primary theoretical ifluece comes from Amartya Se ad Jea Drèze, who locate idividual huma agecy at the heart of developmet. Se s great sythesis work, Developmet as Freedom, cocludes that developmet caot be reduced to material well-beig as evideced by basic icomes, or be achieved by risig average per capita icomes. Rather, it requires a package of overlappig political ad ecoomic mechaisms that progressively eable the exercise of a growig rage of freedoms that allow people to meet their basic eeds ad ulock their iate abilities for self-determiatio. Whe people express their freedom, we call it Agecy. Costituet Voice s other mai itellectual debt is to the semial work of Albert O. Hirschma o the ature of choice uder limited choice coditios. 1 Hirschma observed that whe faced with usatisfactory performace from a orgaizatio, people might decide ot to exit but to ate up to egage to improve the orgaizatio. Hirschma called this Voice. whe faced with usatisfactory performace from a orgaizatio, people might decide ot to exit but to ate up to egage to improve the orgaizatio Our metrics track Agecy ad Voice (amog other thigs) ad show them to be essetial assets for developmet service providers. Costituet Voice method has discovered that feedback loops betwee costituets i developmet ca accurately measure ad cultivate these all-importat persoal ad framig coditios. Because Agecy ad Voice are so cetral to who we are as setiet social beigs, they are both meas ad eds of our social, ecoomic ad political expressios. 1 We have amed the Costituet Voice operatioal cycle after Hirschma. For a further elaboratio of our itellectual debt to Hirschma, see our Feedback Labs blog, Voice As Both Meas Ad Ed. 2

5 Exit, Voice ad Loyalty Developmet as Freedom Theory Costituet Voice Practice Customer Service Participatory Developmet Just as Costituet Voice bleds two itellectual traditios, it also bleds two practice traditios. Both have bee widely utilized for over 60 years but have ever before bee combied. From the world of developmet, Costituet Voice draws o a successio of participatory developmet techiques that emerged out of the field of actio research i the 1950s such as rapid rural appraisal, participatory evaluatio, appreciative equiry ad may others. From the world of cosumer facig busiesses, Costituet Voice draws o the customer service idustry. Customer service metrics trace their roots to the cosumer rights social movemets, which ot coicidetally also emerged i the 1950s. Oe headlie objective of Costituet Voice derives directly from the success of the customer service field. We hypothesize that just as customer service metrics have prove to be reliable predictors of busiess success, Costituet Voice metrics ca predict developmet outcomes. I the world of developmet ad social chage, we have ever doe the work of systematically trackig relatioship metrics ad comparig them to outcome metrics. Costituet Voice is ow doig this work. 3

6 No shortcuts to progress Over the past few years of workig with diverse orgaizatios, Keystoe has leared that there are o shortcuts to the successful cultivatio of Costituet Voice. Every step i the operatioal cycle of Costituet Voice, which we have termed the Hirschma Voice Cycle (pictured below), is ecessary to realize iteded outcomes. Our overridig lesso from the past few years of cliet work is that collectig feedback listeig is ot eough. It is ecessary to lad what you hear i trasparet performace metrics. I order to uderstad the stories behid the measures, it is ecessary to hold a dialogue with your costituets. We call this sese-makig. The you must act o the resultig isight. Fially, ogoig Costituet Voice feedback will let you kow if corrective actios are havig their desired effects. This overall observatio about Costituet Voice is strogly reiforced for us by three commo errors that we have see i orgaizatios that fail i their efforts to improve through feedback from their costituets. The first ad most commo error is to thik that by collectig feedback you have eabled Voice. A broke chai pulls o weight. The most likely fate of collected evaluatio ad moitorig data i developmet today, regrettably, is to lie uderused either aalysed or properly uderstood, ever reported back to cliets or service users, ad rarely acted upo. 5 1 Course correct Affirm stregths, make improvemets ad adjust the feedback mechaism itself Desig Uderstad the theory of chage ad balace rigor, process, cost ad utility The Hirschma Voice Cycle Collect Cotiuous micro-surveys Occasioal i-depth surveys Dialogue Report back ad validate data Coduct targeted ivestigatios Agree plas for improvemets Aalyze Segmet Triagulate Bechmark Visualize 4

7 too much evaluative practice i orgaizatios is cofied to a kid of iteral ghetto, or gulag, that operates o its ow cycles that are too log to be effectively itegrated ito day-to-day maagemet The secod commo error is to over-ivest i feedback data collectio with tools desiged for social research rather tha for performace maagemet. This error typically ivolves goig to the field with log surveys that are costly to implemet ad burdesome for respodets. Moreover, such tools produce overwhelmig volumes of data that staff relegate uused to drawers, boxes ad computer folders. Costituet Voice takes a maagemet perspective o data collectio, that eve imperfect data ca save lives. 2 Oce Costituet Voice is uderstood as a ogoig ad cotiuous process, the stakes with respect to a particular istace of feedback data chage cosiderably. The iitial data collectio does ot have to be perfect as it is subsequetly refied ad validated through dialogue, isight ad improved relatioships. Data Data improvemet Dialogue The third error might be described as ghettoizatio of evaluative activity. Too much evaluative practice i orgaizatios is cofied to a kid of iteral ghetto, or gulag, that operates o its ow cycles that are too log to be effectively itegrated ito day-to-day maagemet. These gulags ted to be led by specialists ad experts who have o lie authority over frotlie workers ad who ted to go outside the orgaizatio to udertake research ad evaluatio activities. Costituet Voice addresses this by embeddig itself directly i ormal maagemet practice. As is explaied below, frotlie workers are expected to be creative evidece-drive problem solvers through two-way commuicatios with costituets. The discussios i this report of each of the steps of the Hirschma Voice Cycle idicate ways to avoid these commo errors. 2 Eve Imperfect Data Ca Save Lives, Suprotik Basu, 9 September impatietoptimists.org/posts/2013/09/progress-o-mdgs-requires-a-obsessio-for-trackig- Results 5

8 Eablig coditios ad readiess Before itroducig Costituet Voice surveys, it is ecessary to establish the miimum coditios for success. The very idea of Costituet Voice is ew for most people. It takes some gettig used to. So the first step to itroducig Costituet Voice at ay orgaizatio is to commuicate the value ad beefit of usig the Costituet Voice cycle i a way that allows people to uderstad ad eve get excited about it. The best ways to do this vary accordig to the costituet group ad seior maagemet eed to craft a approach for iteral ad exteral costituets. Frotlie employees ofte worry about the possibility of egative feedback about their performace. Board may worry that egative feedback will udermie door relatioships. These ad other atural fears eed to be addressed. oly whe they uderstad how it is i their iterest to participate will you ear the kid of survey respose rates that you wat. Similarly, the costituets who are goig to be providig feedback do t have a cotext at first i which to see the value i providig feedback. Oly whe they uderstad how it is i their iterest to participate will you ear the kid of survey respose rates that you wat. I usig o-aoymous feedback, extra care eeds to be used whe explaiig who will see the data, for what purpose, ad i what form. This theme will recur as we discuss data collectio ad also dialogue with feedback providers. Aother dimesio to gettig started with Costituet Voice is to uderstad your orgaizatioal capacity i relatio to the demads of doig Costituet Voice. While Costituet Voice is ot objectively hard to do, our work with orgaizatios over the past few years suggests that doig Costituet Voice does drive orgaizatioal chage. It is extremely helpful to get o the frot foot of this orgaizatioal chage process by buildig a list of the kids of activities that will become secod ature through the itroductio of a Costituet Voice system. For this purpose, we recommed that orgaizatios use the readiess diagostic tool operated by Feedback Labs. 1 1 Feedback Labs is a cosortium of orgaizatios, icludig Keystoe, committed to makig govermets, NGOs ad doors more resposive to the eeds of their costituets 6

9 Testig survey questios to esure their relevace ad saliece for wome i Sidh, Pakista 7

10 Desig Uderstad the theory of chage ad balace rigor, process, cost ad utility Collect Elaboratig the Voice Cycle Stage 1 Desig Course correct 1 Dialogue Aalyze Balacig four measuremet desig priciples Desig is critical. Costituet Voice desig seeks to strike a balace across four cetral priciples of good developmetal measuremet practice: 1 Rigor The data we use to form judgemets ad make decisios should be accurate. We apply good social research ad evaluatio practices to address commo biases i feedback data, icludig those that arise from the various methods we use to collect feedback. These iclude selectio bias, various sample-related biases, ad iterpretatio bias associated with the ucoscious cogitive patters of those desigig ad iterpretig surveys (or focus groups, semi-structured iterviews ad observatioal studies for that matter). 2 Sesitivity to process ad culture Developmet at its best grows the capacity of people to discover solutios ad take cotrol of their lives. It is a livig, geerative process, ad this meas we eed to measure ad urture those thigs that give life to a itervetio: attitudes, relatioships, capabilities, ad agecy. These are steppig-stoes to the chages i material coditios that developmet aims for. Costituets must feel that they belog ad cotribute. Costituet Voice asks questios i a way that is respectful of costituets, for purposes that they uderstad ad edorse. Costituet Voice systems are desiged to build trust ad cofidece betwee a orgaizatio ad its costituets, to geerate kowledge ad isight that costituets ca ad do use for eds that they set, ad, i the process, to grow their capacity. 3 Cost If resources are ifiite, the aythig ca be measured precisely. Resources are fiite, however, ad orgaizatios must make difficult choices betwee moey spet o measuremet-related activities ad moey spet o the itervetio itself. The Costituet Voice cost/value propositio is that for less moey tha covetioal moitorig ad evaluatio, Costituet Voice will deliver more value i terms of cotributio to iteded outcomes. Oe cost advatage of Costituet Voice is the ability to laser i o specific problems, geeratig ideas for solutios, ad sigallig the effects of corrective actios. Aother advatage is that the doig of Costituet Voice has the tedecy to cotribute positively to the itervetio. It is ot desiged to be a eutral measuremet exercise that has o bearig o the thig beig measured. Costituet Voice is desiged to eable the orgaizatio ad its costituets to be more midful of what is happeig ad ot happeig, ad to adjust their behaviors to improve results. 8

11 4 Utility The primary measure of evaluatio has to be its utility. Does it lead to improvemets? Costituet Voice is desiged to geerate data that is useful for the key costituets of a itervetio or orgaizatio icludig the staff ad board of the orgaizatio, the people who are meat to ejoy the value beig created, the fiaciers, the wider field i which the work is located, ad society at large. But first ad foremost, Costituet Voice is desiged to be useful to the orgaizatio udertakig the work. It is iteded to sigificatly ehace relatioships with costituets through more authetic coversatios ad accelerate outcome attaimet. Costituet Voice is a system for cotiuous improvemet i relatioship quality ad geeral performace. It is has a real-time orietatio ad features short measuremet-reflectio-actio cycles. This is implicitly ad explicitly part of the Hirschma Voice Cycle. Fidig the right balace across these four variables is difficult to achieve. I recet times, oe priciple statistical rigor has bee elevated above others as some kid of gold stadard. Gradually, a more sophisticated cosesus is emergig that sees use as the paramout purpose. 3 More practical materials are beig produced to support the use of evaluative fidigs. 4 Rigor Process To strike a balace that is right for a particular itervetio, we start with a careful equiry ito the cotext ad the theory of chage of the itervetio. From there we propose ad test methods ad istrumets to collect feedback. It is essetial to ivolve those who are to be providig feedback at this poit i the desig process. We look for: Utility Costituet Voice Mechaisms of collectig feedback that are affordable, coveiet to the orgaizatio, ad ca exted to large respodet groups; Mechaisms of collectig feedback that are uobjectioable ad o-burdesome to the respodets; Questios uderstood i the same way across the respodet group; Questios usig Likert scales that ca be used over time ad across uits or orgaizatios to eable compariso ad sese-makig dialogues; ad Gettig the right amout of data eough to elighte ad spur further ivestigatio, ot eough to drow us. Cost 3 For a blog makig this poit uequivocally see Time for a Gold Stadard of Use, Fay Twersky, 27 February IterActio i late 2012 produced a four-part guidace ote series o impact evaluatio that cocluded with a ote writte by Keystoe s David Bobright etitled, Use of Impact Evaluatio Results. 9

12 1 Evidece o four dimesios of performace Our work over the past few years helpig orgaizatios to cultivate Costituet Voice has show that Costituet Voice survey methods ca collect reliable evidece of performace across four key ad iterrelated dimesios of orgaizatio performace: It is ecessary to create a bled of questios across the four categories to get the full picture of the respodet s experiece of ad attitudes toward the orgaizatio or itervetio. The relative utility of each category varies for each case depedig o the ature of the work, the cotext, ad the theory of chage. Importace Importace to Respodets Perceptios of outcomes Quality of Service Relatioship Experiece The importace category seeks to establish the importace or relevace of the orgaizatio, service or product to the respodet. Is it cetral to the respodet s life or goals? Or is it relatively trivial? Is it somewhere i betwee? Typical questios iclude: How importat are the services provided by [...] to you ad others i your area? How importat is this issue [or cause] to you? Importace scorig is also used withi a orgaizatio, as exemplified i the rakig schematic opposite: 10

13 11 Quality of Service Questios here hoe i o service quality, ideally as close as possible to a uique touch poit, such as a traiig course, a sales exchage, or a cousellig sessio. Questios here are both geeral ad specific. Examples of specific questios from actual Keystoe surveys iclude: I get the attetio ad support I eed from [ ] staff. We keep busy ad lear somethig i this class every day. Based o the services I receive from [ ], I would recommed [ ] to family or frieds. Have you received this same service from aother orgaisatio? If Yes, how does it compare, much worse, or much better? Did the service happe at a place ad time that was easy for you to atted? Did they have the ecessary resources (teachig materials, equipmet, facilities etc.) to achieve the purpose? Were you able to participate effectively? Please describe the most commo 5 services that you have received from [ ] over the last 12 moths. The rak each activity i terms of how importat it is to you. Activity Importace ratig Not importat < > Extremely importat

14 1 Quality of Relatioship Relatioships matter. Precisely how they matter varies accordig to the ature of the itervetio. Our relatioship questios eable a orgaizatio to determie precisely what their relatioship quality is, ad how to improve it. Overtime, they reveal how relatioship quality affects effectiveess ad outcomes. Our work over the past few years has discovered some core elemets of relatioship quality for most orgaizatios, otably trust, fairess, voice, ad empowermet. These relatioship buildig blocks are iterrelated ad overlappig. By askig related questios, some focussed o specific iteractios ad some more geerally about the relatioship overall, we ca compare aswers ad build up a uaced picture of the orgaizatio s costituet relatioships. Because there is more tha oe side to every relatioship, we also routiely ask respodets to rate their ow readiess to egage the orgaizatio. Commo relatioship questios used i our surveys iclude: Trust (icorporates cofidece, itegrity, maer, credibility, professioal ethics) I have cofidece ad trust i the itegrity of [ ]. I have cofidece i the techical skills of [ ]. Does [...] fulfil its promises - do what it says it will do? Do you feel that workers from [...] really care about you ad wat to help you as best they ca? Does [...] have the ecessary kowledge ad skills to do it well? I feel that [...] is workig sicerely ad hoestly for my beefit. Fairess (icorporates pride ad respect) [ ] treats me fairly. Staff at [ ] always treat me fairly. People at [ ] treat me with courtesy, digity, ad respect. I am proud to be associated with [ ]. 12

15 Voice (icorporates resposiveess) It is worth my effort to egage with [ ] to try to get it to do what I thik is importat. I feel completely free to ask questios ad say what I really thik. I have the opportuities I eed to tell [ ] what I thik about its work. I feel that I cotribute meaigfully to [ ]. I believe that [ ] will satisfactorily respod to ad act upo my feedback. I am sure that [ ] will use my aswers to this survey to improve its services. How much do you ifluece the choice of services offered by [ ]? [...] listes to my questios ad my views ad respods i a geuie way. Empowermet (icorporates agecy ad iclusio) Because of [...], I am more able to stad o your ow feet ad achieve what I wat. Is [...] helpig you to stad o your ow feet, take your ow decisios, fid your ow solutios to problems o your ow? With support from [ ], I am gettig better at achievig my goals. Because of [ ], I have more positive relatioships with other people ad orgaizatios that ca help my life improve. I am more coected to the commuity ad commuity resources thaks to [ ]. Readiess (promotig self-reflectio by the respodet) I feel that I am ready ad willig to try ew thigs offered by [...]. I kow that the more I put ito [...], the more beefits I will get. Outcomes Costituets have a direct ad ofte dispositive experiece of outcomes (or their absece). Costituet perceptios of outcomes are evidece of outcomes that we like to combie with other measures of outcomes to arrive at a composite outcome arrative. This is discussed further uder Aalysis: Triagulatio. With help from [ ], I am makig progress o my goals. I have more icome ad my family lives a better life because of [...]. There are real ad beeficial chages takig place i my life ad my commuity because of [ ]. Are you doig aythig differetly after this receivig the service? Yes/o If yes, what? How well did the [advice/techology/methods/service] work whe you tried them out? [Respose optio icludes, I have ot tried them out ] 13

16 Elaboratig the Voice Cycle Desig Course correct 2 Collect Cotiuous micro-surveys Occasioal i-depth surveys Stage 2 Collect The three most importat thigs to cosider i feedback data collectio are cotext, cotext ad cotext. Dialogue Aalyze Covetioal social sciece-based surveys recko with the complexities of cotext by experimetal methods such as radomizatio across cotrolled comparator groups. If you are prepared to dedicate a large portio of your etire budget, this approach ca tell you what effects you are actually causig. But it will ot geerate cotiuous, real-time data that you ca embed i performace maagemet systems to improve relatioships ad accelerate iteded outcomes. Costituet Voice tackles the complexities of cotext by locatig the process of data collectio as oe step i a five-step cycle that over time geerates higher rates of participatio i surveys ad ever more hoest ad cadid feedback from respodets. Those who practice Costituet Voice uderstad that they must ear their costituets participatio ad cadid feedback. I order to get great feedback quality, they must demostrate to their costituets that good thigs are most likely to happe whe respodets are cadid ad frak i their feedback; i other words, that the effort to provide feedback is rewarded by the improvemets that follow. It is about creatig a culture i which formalized systematic feedback is uderstood by all costituets from staff to society-at-large as a meas to more authetic coversatios about what is workig ad what is ot, how to improve, ad what differet costituets must do i order to realize that improvemet. This culture does ot happe overight, ad ca take time to cultivate. But oce it is there it edures through staff turover ad other forces of chage. those who practice Costituet Voice uderstad that they must ear their costituets participatio ad cadid feedback. Accordigly, the first step i Costituet Voice data collectio is to explai its purpose with iteded respodets. This explaatio typically icludes the followig elemets: A statemet of the purpose of the survey ad how it fits ito a larger purpose of ogoig dialogue ad cotiuous improvemet. A outlie of ad timelie for the steps that will follow from data collectio, icludig reportig back the fidigs of the survey/data collectio method to respodets, ad a commitmet to keep surveyig i order to see if resultig corrective measures are workig. 14

17 Some examples of the kids of thigs that the orgaizatio hopes will happe as a result of the survey. This helps respodets get a tagible sese of the possible beefits of their participatio. Ay explaatios that may be required for the more techical characteristics of data collectio. For example, i the case of a survey, whether the survey resposes are aoymous (ad how that aoymity is protected). A ethical code ad grievace procedure. These framig coditios for a Costituet Voice system are repeated cotiuously as part of a log-term culture buildig ad affirmig process. Aoymous vs. o-aoymous & idepedetly collected vs. self-collected From the orgaizatio s poit of view, o-aoymous data is far more useful tha aoymous data as it ca be matched to other data the orgaizatio has relatig to the respodet. This helps i iterpretig feedback, as well as idetifyig correlatios betwee feedback ad later outcomes. While it is possible to develop some degree of covictio o these thigs by usig characteristics of respodets that do ot iclude their idetity (such as age, geder, legth of relatioship, locatios, etc.), o-aoymous resposes mea that you do ot have to ask these demographic questios i the survey (savig everyoe s time) ad of course the aalysis is far more precise. But from the respodet s poit of view, o the record resposes ca be problematic for differet reasos ragig from ucertaity about the possible egative cosequeces to power dyamics at play i the relatioship. For this reaso, aoymous feedback is geerally less biased tha o-aoymous feedback, ad I m from the compay ad we wat your feedback I order to test courtesy bias, we ra two versios of the same household survey simultaeously across 12 villages i Tazaia. Respodets were selected radomly. The cliet coducted oe survey ad the other survey was coducted by Keystoe. The same itroductio to the survey was give promisig aoymity of resposes. Respodets were aware, however, whe they were takig a survey directly from the cliet. The fidigs from this experimet were strikig. For every questio i which oe could expect to see a courtesy bias (for example, ratig the fairess of the cliet), there was 30 percet more positive respose from the cliet-admiistered survey. For more factual questios or questios where oe would ot expect to see a bias (such as what beefits the respodet received from the compay) the aswers i the two surveys were idistiguishable. 15

18 2 at least iitially is the preferred choice. We say iitially, as whe a orgaizatio has bee gatherig feedback for some time, ad there is a ope culture of hoesty, it is possible to revert to o-aoymous feedback. Similarly, whe a idepedet party with specialized skills collects the data, it is more likely the process will reduce respose biases. From the orgaizatio s poit of view, o the other had, collectig the data themselves may be easier, less costly, ad more quickly embedded i ormal performace maagemet. These dyamics are visualized i the figure below. I it, we assume that bias ad utility are idepedet variables, though of course i practice they are ot. The essetial take-away here is captured i the label variable bias i the figure. This is to say that the bias i the feedback is a factor of the extet to which the respodet uderstads her or his iterest to be best served by givig frak ad hoest feedback. Over time, orgaizatios ca ear this uderstadig i their costituets ad as they do bias goes dow hece the term variable bias. Oe way we have helped our cliets to recogize feedback bias ad work to mitigate it is by coductig parallel data gatherig by ourselves ad by the cliet. The results of this experiece provide the empirical basis for the figure below. Variable Bias High Utility No-aoymous Self- collected Low Bias High Utility Variable Bias Medium Utility Idepedetly collected Aoymous Low Bias Medium Utility 16

19 the bias i the feedback is a factor of the extet to which the respodet uderstads her or his iterest to be best served by givig frak ad hoest feedback. Bi-modal data collectio: light ad heavy Havig decided o a strategy with respect to aoymous vs. o-aoymous ad self-collected vs. idepedetly collected, the ext step is to evisio the overall Costituet Voice system that best fits the orgaizatio ad the steps to build that system, icludig which data collectio methods to use whe. There is o sigle model here. Itroducig a Costituet Voice system requires adaptatio i the orgaizatio s performace maagemet system. This is ever simple ad is usually the mai determiat of the pace of developmet of the Costituet Voice system. Havig emphasized the eed for customizatio ad the fact that Costituet Voice always ivolves orgaizatioal chage, the followig elemets provide a robust framework for work. Cotiuous light-touch feedback data collectio The ideal type ed-state of a Costituet Voice system is oe i which the orgaizatio is gettig feedback cotiuously through micro-surveys ad other techiques at diverse touch poits with its costituets, supplemeted by other feedback collectio that is ot touch poit based. A example of a o-touch poit based data collectio exercise might be a campaig ivitig costituets to call a toll free umber to trigger a short survey or sigal their willigess to be iterviewed by phoe. A wide rage of data collectio mechaisms ca be used, but where surveys are employed they should ot cotai more tha three to five questios. The orgaizatio may have questios i its full list, so these are rotated systematically through the data collectio process to esure a adequate sample of resposes to each questio. Where the data collectio mechaism ad the capabilities ad predilectios of the respodets make it possible (for example, whe respodets are willig ad able to type out aswers), closed Likert questios are combied with a ope questio 17

20 2 that ivites the respodet to explai the reaso for the aswer, or eve to suggest solutios. Resposes to ope questios are aalyzed ad categorized. Give the low cost ad ubiquity of cell phoes ow, text message (SMS), USSD (the techology that drives pay-as-you-go), ad iteractive voice respose (IVR) surveys are becomig a serious optio for may of our cliets. there is cosiderable value to be gleaed from periodic, high-quality, i-depth research. Occasioal i-depth research Eve where micro-ivestigatios are i place, there is cosiderable value to be gleaed from periodic, high-quality, i-depth research. May of our cliets from time to time (typically every moths) supplemet cotiuous micro-surveys with loger surveys admiistered by a idepedet researcher usig rigorous techiques. We have foud that these loger surveys ca help to aswer questios emergig from the micro-surveys, as well as address strategic priorities of the orgaizatio ad set maagemet targets for the future. The comprehesive surveys iclude both quatifiable ratig questios (usig Likert scales) ad ope-eded questios where respodets ca amplify o their scores. Where the surveys are aoymous, selected demographic questios eable disaggregatio ad compariso of resposes. There are additioal importat forms of more i-depth research. These iclude formative ad summative evaluatio studies coducted idepedetly or iterally. Other possibilities are more elaborate observatioal studies, ad participatory research iitiatives. May orgaizatios ca leverage the cosiderable social sciece resources of local uiversities, ofte at o or low cost. 18

21 Data collectio ad research is ot eough The fidigs from cotiuous light-touch surveys ad occasioal i-depth research do ot ormally surface ad develop solutios. They have powerful diagostic value but i themselves are ofte ot eough to guide a full maagemet respose. They sigal where there are issues to address, ad ofte provide clues to actio. The micro-surveys will tell you if corrective measures, oce take, are workig. The developmet of effective solutios ivolves the other steps i the Costituet Voice cycle as described below uder Aalyze ad Dialogue. I Aalyze, we discuss how to compare Costituet Voice data with other evidece available, such as from ormal project moitorig, or from other public data sets. I Dialogue we show how a orgaizatio s frotlie workers ca become the most valuable source of surfacig effective solutios. the micro-surveys will tell you if corrective measures, oce take, are workig. 19

22 Elaboratig the Voice Cycle Desig Stage 3 Aalyze Collect Course correct 3 Dialogue Aalyze Segmet Triagulate Bechmark Visualize So, we have asked the right questios, ad collected valid aswers, ow we come to aalyze ad iterpret the data. Part of iterpretatio comes through dialogue with costituets to create a shared uderstadig of what the data ad aalysis may really mea. This sese-makig aspect of iterpretatio is discussed i the ext sectio. Quatified perceptual data ca be aalysed usig stadard statistical methods to give actioable isight ito the perceptios of differet groups of costituets, ad that by trackig these measures of perceptios ad their aalyses over time it is possible to refie ever more powerful isight. The emphasis i our aalytics is to geerate clear coclusios ad to represet those fidigs i simple graphics. I additio to descriptive statistics, we employ three mai types of aalysis segmetatio, triagulatio ad bechmarkig. We discuss each of these below. Segmet Costituet Voice aalysis typically begis by segmetig resposes i two ways: by idividual characteristics ad by creatig sub-groups of costituets (or clusters) depedig o their resposes. The geeral purpose here is to be able to use the aalysis i order to develop corrective actios ad strategies that fit with differet segmets of a orgaizatio s costituets. Idividual characteristics To start, the cliet sets the importace to it for the idividual (or, i the trade, demographic) characteristics of the respodets, for example, sex, age, ature of relatioship with the cliet (occasioal, frequet), locatio. By disaggregatig aswers to questios by demographic characteristics oe ca clearly see relative success ad failure i coectig across the etire group of costituets. Addressig the geder gaps I Keystoe s Developmet Parterships Survey we are fidig out that, eve after cosiderable efforts by our cliets to apply a geder approach, female respodets ted to report that they receive less capacity buildig support tha male respodets. This poits out that may INGOs eed to redouble their efforts to cosult with wome ad esure their participatio i capacity buildig support iitiatives. We have recommeded that cliets stregthe moitorig metrics for wome s participatio i capacity buildig ad goig forward we will compare this project moitorig data with wome s feedback. 20

23 Clusterig The other way is by clusterig respodets accordig to their resposes to certai questios i order to create categories that are more useful for developig ad executig corrective actios. Whe we use the familiar 0 to 10 Likert scale i customer service surveys, we also use a approach to clusterig called et promoter aalysis (NPA). 5 NPA is disarmigly simple, but it has prove to be a reliable measure of customer loyalty, ad a powerful lever for positive orgaizatioal motivatio ad chage. As illustrated i the figure below, NPA classifies respodets ito promoters, passives ad detractors ad calculates a sigle et promoter score (NP Score). It may seem to some people that the divisio of respodets i these three groups is somewhat arbitrary. If we have for example a scale of 0- absolutely do t agree to 10- absolutely agree, the why do we cosider that someoe givig a ratig of 6 is a detractor? What the customer service idustry has leared over time ad what we are seeig i the social value creatig sector, is that this accommodates the courtesy bias i survey resposes. Empirical evidece shows that people givig ratigs i the middle or just above the middle of the scale are ormally uderstatig their dissatisfactio. The coverse is ot show to be true. Those givig lower scores are ot uderstatig their satisfactio. I ay case, if your goal is excellece, the you ca ever be satisfied with aythig below a 9. 5 Net Promoter is a registered trademark of Fred Reichheld, Bai & Compay ad Satmetrix. For more see: as well as the ope source et promoter commuity at 21

24 3 All Likert scales also led themselves to clusterig. The task is to combie scores i a way that creates groups that for the purpose of corrective actio are likely to react similarly. For certai kids of surveys (e.g., USSD or IVR), smaller scales are more appropriate ad i some cases scale descriptors ca allow respodets to idetify themselves with a particular attitude, such as doubter, fece-sitter, qualified supporter, uqualified supporter. Oe importat sub-group are those who do ot respod to surveys. Costituet Voice places a high value o respose rates. Higher participatio rates, while ot always guarateeig represetatio, reduce the likelihood of a sigificat respose bias i the data. We track ad report respose rates as a key idicator of relatioship quality. The compaies that ivest most seriously i customer satisfactio metrics do this. Some customer-cetric compaies eve put o-respoders ito the detractor category whe calculatig et promoter scores. This pushes NP scores dow, forcig them to work o wiig high participatio i feedback loops. This is discussed further uder Dialogue. Because Costituet Voice is a ogoig process of egagemet, over time we expect to see higher ad higher respose rates as costituets realize that great thigs happe whe they provide frak ad hoest feedback. I order to uderstad more about sub-clusters, we routiely compare aswers of two or more questios to determie the patters of distributio of particular aswers. So, we would ask, how do resposes to a fairess questio ([ ] treats me fairly) compare with resposes to a outcomes questio (I have more icome ad my family lives a better life because of [...]). Applyig stadard statistical tests, we assig a statistical sigificace of ay variatios from what we would expect to see. For example, we might fid o sigificat correlatios for wome as distict from me with respect to either the fairess or outcomes questios whe looked at idepedetly. But whe we do a geder disaggregatio for how all detractors to the fairess questio respod to the outcomes questio, we might see (i fact have see) that wome report fewer beefits. These kids of fidigs geerate questios that are explored further i Dialogue activities with respodets. The Meaig of satisfactio A survey with tree platatio workers showed a uexpected patter. Workers gave high scores o satisfactio with the compay ad low scores o the compay treats me fairly. Whe asked to explai these seemigly cotradictory aswers, the workers made it clear that their aswers to the satisfactio questio idicated that they eeded the job, as they mostly had o alterative for wage employmet. But, they added, this had othig to do with how the compay treated them, which they felt was too ofte ufair. By further equiry we were able to surface specific grievaces that, oce addressed, resulted i a sigificat drop i arso i the compay s forests. 22

25 it is idetifyig patters of resposes that may show us previously hidde factors that are shapig results. We are ot claimig that this kid of statistical aalysis is geeratig defiitive aswers. Rather, it is idetifyig patters of resposes that may show us previously hidde factors that are shapig results. By teasig these out, we ca use them i sese-makig activities with costituets ad more geerally i the ogoig iteractios betwee the orgaizatio ad its costituets. We wat to stregthe relatioships to get better results for costituets ad for the orgaizatio, so the focus is always o testig actios to improve. Havig discovered, for example, that 30 percet of those we serve are cosistetly detractors, we ca we address their specific cocers. The data idetify them for us, ad they will tell us whe we have succeeded with them. Data throw up questios or hypotheses; we tur these ito a process of shared equiry with costituets. The operative word here is we, ad the data tell us much more tha we kew before about the we. Triagulate Triagulatio is a facy word for comparative aalysis, usually drawig from differet data sources or usig differet methods of research (classically, quatitative vs. qualitative). For example, i-depth iterviews could be coducted with certai groups to gai deeper isight ito their perspectives o program outcomes. These perspectives are the compared with the fidigs from survey questios o outcomes. Puttig the two together uderwrites higher reliability of the fidigs. A primary objective i much of our cliet work is to discover cosistet correlatios betwee real time perceptual data from our surveys ad later occurrig outcomes measured idepedetly (ad preferably objectively). The customer service idustry provides the most famous example of this. After fifty years ad iumerable studies, o oe doubts that customer loyalty properly measured i surveys today is a accurate predictor of profits, shareholder value ad corporate growth. Orgaizatios who triagulate perceptual feedback data with outcome data are begiig to fid some iterestig correlatios. Oe Keystoe cliet who has begu comparig the two sources of data have foud that promoters are twice as likely to achieve a iteded outcome 23

26 3 25 Promoters are twice as likely as those who are ucoviced to achieve a goal or a outcome Percet 0 Ucoviced (0-6) Neutrals (7-8) Promoters (9-10) tha those givig low scores. (see figure above). By askig a hadful of very short questios regularly, the orgaisatio is able to idetify today those of its costituets who abset some chage are ot goig to achieve iteded outcomes i the future. Based o these metrics, the orgaisatio is providig extra support to these idividuals. Triagulatio is provig to be a powerful tool, helpig aticipate outcomes as well as eablig uderstadig o how to improve programs ad itervetios to maximize impact. We are also seeig some excitig predictive idicators emergig i other areas of our work. Studet feedback correlates to the scores they later receive o stadardized tests. Worker feedback correlates to abseteeism, accidets, grievace rates ad icidets of sabotage. Smallholder farmer feedback aticipates rates of adoptio of ew techologies ad subsequet crop yields. Resposes to a questio about coectios to commuity predict how quickly a very low icome perso progresses i attaiig fiacial goals. Feedback from displaced people livig i camps poits to the obstacles to their self-reliace ad how to remove them. Predictive idicators are oe of the best tools available to achieve better results more quickly, particularly whe combied with bechmarkig (as discussed below). I additio, feedback from costituets ca be compared to other metrics such as fiacial performace, cesus data or data from other sources. I a youth employmet project for example we may cross feedback from youths expressig that they perceive 24

27 Work by day, bur by ight I the same case of the tree platatio compay, the compay gaied cofidece i the fidigs from the survey whe we compared the survey fidigs disaggregated by village with the compay s ow records of harm to its forests. The table below shows that there was little harm to the forest aroud the villages that reported more positive attitudes to the compay, while those that reported more egative scores were located ext to areas of sigificat harm to the forest. Huma Impact o forest attributed to earest village Correlatio of commuity impacts o compay forests with commuity perceptios of the compay 200 Village A Village C Village B Village F 50 Village D E Village F Village G H Village Commuity perceptios by village themselves to be more employable with data from the local employmet office idicatig a icrease i applicatios to social isurace for people i the same age rage. Bechmark Quatificatio of qualitative feedback data eables us to create performace bechmarks ot o the basis of techical ratigs by exteral ispectors, but o feedback from those i the best positio to speak of their experiece of a orgaizatio or service. Comparable feedback data are geerated whe the same questios are asked i the same ways across groups or at differet times. Some orgaizatios compare their ow performace over time or across uits but oly rarely do they do so agaist other orgaizatios doig similar thigs. Bechmarkig deepes your uderstadig of what a particular umerical aswer meas by showig it i relatio to other scores. By comparig agaist the average, you uderstad what ormal meas. Ad by comparig agaist high-performig 25

28 3 outliers, your imagiatio of what is possible is stretched. Say the maager i Regio A sees that his regio received a et promoter score of 25. He might be cotet with this util he sees that Regios B-G have a average NP score of 37. A close look at Regios B-G also reveals that the top scorig Regio F has a NP score of 65, icludig oly 2 percet detractors. The maager i Regio A ow kows what top performers ca do to lift scores, ad ca ask the maager i Regio F how she did it. I additio, whe a critical mass of orgaizatios start to pool their feedback data ad bechmark agaist each other, we ca idetify treds i the sector as well as extract best practices that ca be used for improvig performace collectively. Keystoe offers a data sharig opportuity to cliets to eable them to compare their data ad beefit from field level isight derivig from their aggregated feedback data. I partership with leadig gratmakig foudatios ad Feedback Labs, Keystoe has created the Feedback Commos as a olie platform where orgaizatios ca share ad compare their feedback data. Visualize There are a growig selectio of sophisticated ad easily available tools that ca help visually preset data. Visualizatios ca help deepe uderstadig of the data, ad idetify meaig withi it. For example, geo-mappig tools plot data across a geographical area. The example visualizatio below plots scores withi a geographical area of a refugee camp. It ucovers a cocetratio of low scores (show i red) i a particular area that is hard to igore! Such tools are particularly useful i dialogue sessios with respodets. 26

29 Towards idexes Net promote aalysis is oe way to reduce complex patters to simple, yet powerful sigle metrics. Aother techique is idexig. We have foud that at times it ca be useful to combie resposes to questios ito a sigle idex that is tracked ad compared over time. The value of a idex is i its ability to represet separate but related elemets i a sigle umber. Reducig data ito bite-size pieces helps orgaizatios uderstad it ad respod to it. The more we kow about how idividual questios correlate to performace ad results (as discussed above uder Segmetatio ), the more explaatory the idex. We have foud it particularly useful to aggregate very specific micro-questios relatig to service quality at a specific touch poit ito a Service Quality Idex, for example. Whe differet services offered by a orgaizatio are idexed i this way, it is meaigful to compare aggregate service quality across these services. Similarly, by combiig Relatioship Quality questios we achieve a Relatioship Idex that cliets have foud to be a compellig motivator for staff to implemet reforms. 27

30 Elaboratig the Voice Cycle Desig Stage 4 Dialogue Collect Course correct 4 It s ot about the data, stupid. Dialogue Report back ad validate data Coduct targeted ivestigatios Agree plas for improvemets Aalyze The sigle most importat step i the cycle ivolves closig the loop by reportig back to costituets what you heard ad co-creatig solutios. We call this activity sese-makig, as i makig sese of the fidigs through ope coversatios with respodets that geerate a shared uderstadig of ways to improve. Two thigs happe whe you ask costituets questios. Oe is you get aswers. The other is that you raise expectatios. By holdig a dialogue with respodets you do two thigs. You refie ad validate the meaig of the aswers to the questios. Ad you maage the expectatios created by askig the questios by emergig solutios that are withi the actual capabilities of the orgaizatio ad the respodets. I the place of ubouded expectatios, you ow have a two-way dialogic process that grouds costituet expectatios i graular data This is what we heard you sayig. This is what we propose to do. What do you thik of that? Ad by the way, these are some of the factors that costrai our capacity to respod here. What ca you cotribute to a solutio here? Orgaizatios that move beyod data collectio to dialogue ot oly lear ad improve their performace, they uderwrite higher respose rates ad more frak feedback i future surveys because costituets clearly see the value i providig feedback. I fact, this Dialogue stage is crucial for ivestig i the feedback mechaism. By clearly demostratig that feedback is beig listeed to ad respoded to appropriately, a orgaizatio will yield higher resposes ad eve better feedback data as it cotiues to repeat the cycle. The ext stage Course correct will provide the opportuity to tweak the feedback mechaism based o the discussios that happe durig this Dialogue stage. 6 6 A example from a Ugada program of commuity moitorig of rural health cliics illustrates this geeral poit i a fasciatig way. A experimetal study has foud that [e]fforts to stimulate beeficiary cotrol, coupled with the provisio of report cards o staff performace, resulted i sigificat improvemets i health care delivery ad health outcomes i both the short ad the loger ru. Efforts to stimulate beeficiary cotrol without providig iformatio o performace had o impact o quality of care or health outcomes. ( org/exteral/default/mai?pagepk= &thesitepk=469382&pipk= &meu PK= &etityID= _ &cid=DEC_PolicyResearchEN_D_INT; dowloaded 5 September 2014). I other words, whe costituets were give iformatio about orgaizatioal performace, the whole commuity moitorig effort was more effective i geeratig improvemets i outcomes. This is a origial ad illustrative isight ito how to ivest i the feedback mechaism. 28

Consortium: North Carolina Community Colleges

Consortium: North Carolina Community Colleges Associatio of Research Libraries / Texas A&M Uiversity www.libqual.org Cotributors Collee Cook Texas A&M Uiversity Fred Heath Uiversity of Texas BruceThompso Texas A&M Uiversity Martha Kyrillidou Associatio

More information

part2 Participatory Processes

part2 Participatory Processes part part2 Participatory Processes Participatory Learig Approaches Whose Learig? Participatory learig is based o the priciple of ope expressio where all sectios of the commuity ad exteral stakeholders

More information

Application for Admission

Application for Admission Applicatio for Admissio Admissio Office PO Box 2900 Illiois Wesleya Uiversity Bloomig, Illiois 61702-2900 Apply o-lie at: www.iwu.edu Applicatio Iformatio I am applyig: Early Actio Regular Decisio Early

More information

HANDBOOK. Career Center Handbook. Tools & Tips for Career Search Success CALIFORNIA STATE UNIVERSITY, SACR AMENTO

HANDBOOK. Career Center Handbook. Tools & Tips for Career Search Success CALIFORNIA STATE UNIVERSITY, SACR AMENTO HANDBOOK Career Ceter Hadbook CALIFORNIA STATE UNIVERSITY, SACR AMENTO Tools & Tips for Career Search Success Academic Advisig ad Career Ceter 6000 J Street Lasse Hall 1013 Sacrameto, CA 95819-6064 916-278-6231

More information

E-LEARNING USABILITY: A LEARNER-ADAPTED APPROACH BASED ON THE EVALUATION OF LEANER S PREFERENCES. Valentina Terzieva, Yuri Pavlov, Rumen Andreev

E-LEARNING USABILITY: A LEARNER-ADAPTED APPROACH BASED ON THE EVALUATION OF LEANER S PREFERENCES. Valentina Terzieva, Yuri Pavlov, Rumen Andreev Titre du documet / Documet title E-learig usability : A learer-adapted approach based o the evaluatio of leaer's prefereces Auteur(s) / Author(s) TERZIEVA Valetia ; PAVLOV Yuri (1) ; ANDREEV Rume (2) ;

More information

Fuzzy Reference Gain-Scheduling Approach as Intelligent Agents: FRGS Agent

Fuzzy Reference Gain-Scheduling Approach as Intelligent Agents: FRGS Agent Fuzzy Referece Gai-Schedulig Approach as Itelliget Agets: FRGS Aget J. E. ARAUJO * eresto@lit.ipe.br K. H. KIENITZ # kieitz@ita.br S. A. SANDRI sadra@lac.ipe.br J. D. S. da SILVA demisio@lac.ipe.br * Itegratio

More information

arxiv: v1 [cs.dl] 22 Dec 2016

arxiv: v1 [cs.dl] 22 Dec 2016 ScieceWISE: Topic Modelig over Scietific Literature Networks arxiv:1612.07636v1 [cs.dl] 22 Dec 2016 A. Magalich, V. Gemmetto, D. Garlaschelli, A. Boyarsky Uiversity of Leide, The Netherlads {magalich,

More information

Natural language processing implementation on Romanian ChatBot

Natural language processing implementation on Romanian ChatBot Proceedigs of the 9th WSEAS Iteratioal Coferece o SIMULATION, MODELLING AND OPTIMIZATION Natural laguage processig implemetatio o Romaia ChatBot RALF FABIAN, MARCU ALEXANDRU-NICOLAE Departmet for Iformatics

More information

'Norwegian University of Science and Technology, Department of Computer and Information Science

'Norwegian University of Science and Technology, Department of Computer and Information Science The helpful Patiet Record System: Problem Orieted Ad Kowledge Based Elisabeth Bayega, MS' ad Samso Tu, MS2 'Norwegia Uiversity of Sciece ad Techology, Departmet of Computer ad Iformatio Sciece ad Departmet

More information

Management Science Letters

Management Science Letters Maagemet Sciece Letters 4 (24) 2 26 Cotets lists available at GrowigSciece Maagemet Sciece Letters homepage: www.growigsciece.com/msl A applicatio of data evelopmet aalysis for measurig the relative efficiecy

More information

2014 Gold Award Winner SpecialParent

2014 Gold Award Winner SpecialParent Award Wier SpecialParet Dedicated to all families of childre with special eeds 6 th Editio/Fall/Witer 2014 Desig ad Editorial Awards Competitio MISSION Our goal is to provide parets of childre with special

More information

VISION, MISSION, VALUES, AND GOALS

VISION, MISSION, VALUES, AND GOALS 6 VISION, MISSION, VALUES, AND GOALS 2010-2015 VISION STATEMENT Ohloe College will be kow throughout Califoria for our iclusiveess, iovatio, ad superior rates of studet success. MISSION STATEMENT The Missio

More information

also inside Continuing Education Alumni Authors College Events

also inside Continuing Education Alumni Authors College Events SUMMER 2016 JAMESTOWN COMMUNITY COLLEGE ALUMNI MAGAZINE create a etrepreeur creatig a busiess a artist creatig beauty a citize creatig the future also iside Cotiuig Educatio Alumi Authors College Evets

More information

On March 15, 2016, Governor Rick Snyder. Continuing Medical Education Becomes Mandatory in Michigan. in this issue... 3 Great Lakes Veterinary

On March 15, 2016, Governor Rick Snyder. Continuing Medical Education Becomes Mandatory in Michigan. in this issue... 3 Great Lakes Veterinary michiga veteriary medical associatio i this issue... 3 Great Lakes Veteriary Coferece 4 What You Need to Kow Whe Issuig a Iterstate Certificate of Ispectio 6 Low Pathogeic Avia Iflueza H5 Virus Detectios

More information

Delaware Performance Appraisal System Building greater skills and knowledge for educators

Delaware Performance Appraisal System Building greater skills and knowledge for educators Delaware Performance Appraisal System Building greater skills and knowledge for educators DPAS-II Guide for Administrators (Assistant Principals) Guide for Evaluating Assistant Principals Revised August

More information

AUTHORITATIVE SOURCES ADULT AND COMMUNITY LEARNING LEARNING PROGRAMMES

AUTHORITATIVE SOURCES ADULT AND COMMUNITY LEARNING LEARNING PROGRAMMES AUTHORITATIVE SOURCES ADULT AND COMMUNITY LEARNING LEARNING PROGRAMMES AUGUST 2001 Contents Sources 2 The White Paper Learning to Succeed 3 The Learning and Skills Council Prospectus 5 Post-16 Funding

More information

On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC

On Human Computer Interaction, HCI. Dr. Saif al Zahir Electrical and Computer Engineering Department UBC On Human Computer Interaction, HCI Dr. Saif al Zahir Electrical and Computer Engineering Department UBC Human Computer Interaction HCI HCI is the study of people, computer technology, and the ways these

More information

Probability estimates in a scenario tree

Probability estimates in a scenario tree 101 Chapter 11 Probability estimates in a scenario tree An expert is a person who has made all the mistakes that can be made in a very narrow field. Niels Bohr (1885 1962) Scenario trees require many numbers.

More information

National and Regional performance and accountability: State of the Nation/Region Program Costa Rica.

National and Regional performance and accountability: State of the Nation/Region Program Costa Rica. National and Regional performance and accountability: State of the Nation/Region Program Costa Rica. Miguel Gutierrez Saxe. 1 The State of the Nation Report: a method to learn and think about a country.

More information

MARKETING FOR THE BOP WORKSHOP

MARKETING FOR THE BOP WORKSHOP MARKETING FOR THE BOP WORKSHOP Concept Note This note presents our methodology to help refine the marketing and sales practices of organizations that sell innovative devices (such as water filters or improved

More information

Market Intelligence. Alumni Perspectives Survey Report 2017

Market Intelligence. Alumni Perspectives Survey Report 2017 Market Intelligence Alumni Perspectives Survey Report 2017 Contents Executive Summary... 2 Introduction.... 5 Key Findings... 6 The Value of a Graduate Management Education.... 8 Three Dimensions of Value....

More information

Dakar Framework for Action. Education for All: Meeting our Collective Commitments. World Education Forum Dakar, Senegal, April 2000

Dakar Framework for Action. Education for All: Meeting our Collective Commitments. World Education Forum Dakar, Senegal, April 2000 Dakar Framework for Action Education for All: Meeting our Collective Commitments Text adopted by the World Education Forum Dakar, Senegal, 26-28 April 2000 Dakar Framework for Action Education for All:

More information

WORK OF LEADERS GROUP REPORT

WORK OF LEADERS GROUP REPORT WORK OF LEADERS GROUP REPORT ASSESSMENT TO ACTION. Sample Report (9 People) Thursday, February 0, 016 This report is provided by: Your Company 13 Main Street Smithtown, MN 531 www.yourcompany.com INTRODUCTION

More information

UPPER SECONDARY CURRICULUM OPTIONS AND LABOR MARKET PERFORMANCE: EVIDENCE FROM A GRADUATES SURVEY IN GREECE

UPPER SECONDARY CURRICULUM OPTIONS AND LABOR MARKET PERFORMANCE: EVIDENCE FROM A GRADUATES SURVEY IN GREECE UPPER SECONDARY CURRICULUM OPTIONS AND LABOR MARKET PERFORMANCE: EVIDENCE FROM A GRADUATES SURVEY IN GREECE Stamatis Paleocrassas, Panagiotis Rousseas, Vassilia Vretakou Pedagogical Institute, Athens Abstract

More information

2015 Annual Report to the School Community

2015 Annual Report to the School Community 2015 Annual Report to the School Community Narre Warren South P-12 College School Number: 8839 Name of School Principal: Rob Duncan Name of School Council President: Greg Bailey Date of Endorsement: 23/03/2016

More information

Major Milestones, Team Activities, and Individual Deliverables

Major Milestones, Team Activities, and Individual Deliverables Major Milestones, Team Activities, and Individual Deliverables Milestone #1: Team Semester Proposal Your team should write a proposal that describes project objectives, existing relevant technology, engineering

More information

BENCHMARK TREND COMPARISON REPORT:

BENCHMARK TREND COMPARISON REPORT: National Survey of Student Engagement (NSSE) BENCHMARK TREND COMPARISON REPORT: CARNEGIE PEER INSTITUTIONS, 2003-2011 PREPARED BY: ANGEL A. SANCHEZ, DIRECTOR KELLI PAYNE, ADMINISTRATIVE ANALYST/ SPECIALIST

More information

Redirected Inbound Call Sampling An Example of Fit for Purpose Non-probability Sample Design

Redirected Inbound Call Sampling An Example of Fit for Purpose Non-probability Sample Design Redirected Inbound Call Sampling An Example of Fit for Purpose Non-probability Sample Design Burton Levine Karol Krotki NISS/WSS Workshop on Inference from Nonprobability Samples September 25, 2017 RTI

More information

B. How to write a research paper

B. How to write a research paper From: Nikolaus Correll. "Introduction to Autonomous Robots", ISBN 1493773070, CC-ND 3.0 B. How to write a research paper The final deliverable of a robotics class often is a write-up on a research project,

More information

Worldwide Online Training for Coaches: the CTI Success Story

Worldwide Online Training for Coaches: the CTI Success Story Worldwide Online Training for Coaches: the CTI Success Story Case Study: CTI (The Coaches Training Institute) This case study covers: Certification Program Professional Development Corporate Use icohere,

More information

University of Waterloo School of Accountancy. AFM 102: Introductory Management Accounting. Fall Term 2004: Section 4

University of Waterloo School of Accountancy. AFM 102: Introductory Management Accounting. Fall Term 2004: Section 4 University of Waterloo School of Accountancy AFM 102: Introductory Management Accounting Fall Term 2004: Section 4 Instructor: Alan Webb Office: HH 289A / BFG 2120 B (after October 1) Phone: 888-4567 ext.

More information

Harvesting the Wisdom of Coalitions

Harvesting the Wisdom of Coalitions Harvesting the Wisdom of Coalitions Understanding Collaboration and Innovation in the Coalition Context February 2015 Prepared by: Juliana Ramirez and Samantha Berger Executive Summary In the context of

More information

Critical Thinking in Everyday Life: 9 Strategies

Critical Thinking in Everyday Life: 9 Strategies Critical Thinking in Everyday Life: 9 Strategies Most of us are not what we could be. We are less. We have great capacity. But most of it is dormant; most is undeveloped. Improvement in thinking is like

More information

Early Warning System Implementation Guide

Early Warning System Implementation Guide Linking Research and Resources for Better High Schools betterhighschools.org September 2010 Early Warning System Implementation Guide For use with the National High School Center s Early Warning System

More information

STEPS TO EFFECTIVE ADVOCACY

STEPS TO EFFECTIVE ADVOCACY Poverty, Conservation and Biodiversity Godber Tumushabe Executive Director/Policy Analyst Advocates Coalition for Development and Environment STEPS TO EFFECTIVE ADVOCACY UPCLG Advocacy Capacity Building

More information

Why Pay Attention to Race?

Why Pay Attention to Race? Why Pay Attention to Race? Witnessing Whiteness Chapter 1 Workshop 1.1 1.1-1 Dear Facilitator(s), This workshop series was carefully crafted, reviewed (by a multiracial team), and revised with several

More information

MGT/MGP/MGB 261: Investment Analysis

MGT/MGP/MGB 261: Investment Analysis UNIVERSITY OF CALIFORNIA, DAVIS GRADUATE SCHOOL OF MANAGEMENT SYLLABUS for Fall 2014 MGT/MGP/MGB 261: Investment Analysis Daytime MBA: Tu 12:00p.m. - 3:00 p.m. Location: 1302 Gallagher (CRN: 51489) Sacramento

More information

TU-E2090 Research Assignment in Operations Management and Services

TU-E2090 Research Assignment in Operations Management and Services Aalto University School of Science Operations and Service Management TU-E2090 Research Assignment in Operations Management and Services Version 2016-08-29 COURSE INSTRUCTOR: OFFICE HOURS: CONTACT: Saara

More information

DERMATOLOGY. Sponsored by the NYU Post-Graduate Medical School. 129 Years of Continuing Medical Education

DERMATOLOGY. Sponsored by the NYU Post-Graduate Medical School. 129 Years of Continuing Medical Education Advaces i DERMATOLOGY THURSDAY - FRIDAY JUNE 7-8, 2012 New York, NY Sposored by the NYU Post-Graduate Medical School 129 Years of Cotiuig Medical Educatio THE RONALD O. PERELMAN DEPARTMENT OF DERMATOLOGY

More information

Introduction. 1. Evidence-informed teaching Prelude

Introduction. 1. Evidence-informed teaching Prelude 1. Evidence-informed teaching 1.1. Prelude A conversation between three teachers during lunch break Rik: Barbara: Rik: Cristina: Barbara: Rik: Cristina: Barbara: Rik: Barbara: Cristina: Why is it that

More information

Superintendent s 100 Day Entry Plan Review

Superintendent s 100 Day Entry Plan Review Superintendent s 100 Day Entry Plan Review January 10, 2012 Thelma Meléndez de Santa Ana, Ph.D. Superintendent 2 The 100-Day Entry Plan Roll-Out What We ll Cover Reflections & Observations on Our Aha!

More information

A Note on Structuring Employability Skills for Accounting Students

A Note on Structuring Employability Skills for Accounting Students A Note on Structuring Employability Skills for Accounting Students Jon Warwick and Anna Howard School of Business, London South Bank University Correspondence Address Jon Warwick, School of Business, London

More information

ABET Criteria for Accrediting Computer Science Programs

ABET Criteria for Accrediting Computer Science Programs ABET Criteria for Accrediting Computer Science Programs Mapped to 2008 NSSE Survey Questions First Edition, June 2008 Introduction and Rationale for Using NSSE in ABET Accreditation One of the most common

More information

The Round Earth Project. Collaborative VR for Elementary School Kids

The Round Earth Project. Collaborative VR for Elementary School Kids Johnson, A., Moher, T., Ohlsson, S., The Round Earth Project - Collaborative VR for Elementary School Kids, In the SIGGRAPH 99 conference abstracts and applications, Los Angeles, California, Aug 8-13,

More information

The Use of Statistical, Computational and Modelling Tools in Higher Learning Institutions: A Case Study of the University of Dodoma

The Use of Statistical, Computational and Modelling Tools in Higher Learning Institutions: A Case Study of the University of Dodoma International Journal of Computer Applications (975 8887) The Use of Statistical, Computational and Modelling Tools in Higher Learning Institutions: A Case Study of the University of Dodoma Gilbert M.

More information

An Empirical Analysis of the Effects of Mexican American Studies Participation on Student Achievement within Tucson Unified School District

An Empirical Analysis of the Effects of Mexican American Studies Participation on Student Achievement within Tucson Unified School District An Empirical Analysis of the Effects of Mexican American Studies Participation on Student Achievement within Tucson Unified School District Report Submitted June 20, 2012, to Willis D. Hawley, Ph.D., Special

More information

Higher education is becoming a major driver of economic competitiveness

Higher education is becoming a major driver of economic competitiveness Executive Summary Higher education is becoming a major driver of economic competitiveness in an increasingly knowledge-driven global economy. The imperative for countries to improve employment skills calls

More information

Alternative education: Filling the gap in emergency and post-conflict situations

Alternative education: Filling the gap in emergency and post-conflict situations 2009 Alternative education: Filling the gap in emergency and post-conflict situations Overview Children and youth have a wide range of educational needs in emergency situations, especially when affected

More information

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur

Module 12. Machine Learning. Version 2 CSE IIT, Kharagpur Module 12 Machine Learning 12.1 Instructional Objective The students should understand the concept of learning systems Students should learn about different aspects of a learning system Students should

More information

School Leadership Rubrics

School Leadership Rubrics School Leadership Rubrics The School Leadership Rubrics define a range of observable leadership and instructional practices that characterize more and less effective schools. These rubrics provide a metric

More information

5th Grade Unit Plan Social Studies Comparing the Colonies. Created by: Kylie Daniels

5th Grade Unit Plan Social Studies Comparing the Colonies. Created by: Kylie Daniels 5th Grade Unit Plan Social Studies Comparing the Colonies Created by: Kylie Daniels 1 Table of Contents Unit Overview pp. 3 7 Lesson Plan 1 pp. 8 11 Lesson Plan 2 pp. 12 15 Lesson Plan 3 pp. 16 19 Lesson

More information

Kaipaki School. We expect the roll to climb to almost 100 in line with the demographic report from MoE through 2016.

Kaipaki School. We expect the roll to climb to almost 100 in line with the demographic report from MoE through 2016. Kaipaki School 687 Kaipaki Rd RD3 Cambridge Kaipaki School Bringing Learning to Life Whakatinanahia te mātauranga Ph: (07) 823 6653 e-mail: principal@kaipaki.school.nz www.kaipaki.school.nz 25 May 2015

More information

Queensborough Public Library (Queens, NY) CCSS Guidance for TASC Professional Development Curriculum

Queensborough Public Library (Queens, NY) CCSS Guidance for TASC Professional Development Curriculum CCSS Guidance for TASC Professional Development Curriculum Queensborough Public Library (Queens, NY) DRAFT Version 1 5/19/2015 CCSS Guidance for NYSED TASC Curriculum Development Background Victory Productions,

More information

EQuIP Review Feedback

EQuIP Review Feedback EQuIP Review Feedback Lesson/Unit Name: On the Rainy River and The Red Convertible (Module 4, Unit 1) Content Area: English language arts Grade Level: 11 Dimension I Alignment to the Depth of the CCSS

More information

The Condition of College & Career Readiness 2016

The Condition of College & Career Readiness 2016 The Condition of College and Career Readiness This report looks at the progress of the 16 ACT -tested graduating class relative to college and career readiness. This year s report shows that 64% of students

More information

ADAPTIVE PLANNING. 1 Powered by POeT Solvers Limited

ADAPTIVE PLANNING. 1  Powered by POeT Solvers Limited ADAPTIVE PLANNING 1 www.pmtutor.org Powered by POeT Solvers Limited ADAPTIVE PLANNING Adaptive planning is the conscious acceptance that early plans are both necessary and likely to be flawed; therefore,

More information

SETTING STANDARDS FOR CRITERION- REFERENCED MEASUREMENT

SETTING STANDARDS FOR CRITERION- REFERENCED MEASUREMENT SETTING STANDARDS FOR CRITERION- REFERENCED MEASUREMENT By: Dr. MAHMOUD M. GHANDOUR QATAR UNIVERSITY Improving human resources is the responsibility of the educational system in many societies. The outputs

More information

General study plan for third-cycle programmes in Sociology

General study plan for third-cycle programmes in Sociology Date of adoption: 07/06/2017 Ref. no: 2017/3223-4.1.1.2 Faculty of Social Sciences Third-cycle education at Linnaeus University is regulated by the Swedish Higher Education Act and Higher Education Ordinance

More information

VIEW: An Assessment of Problem Solving Style

VIEW: An Assessment of Problem Solving Style 1 VIEW: An Assessment of Problem Solving Style Edwin C. Selby, Donald J. Treffinger, Scott G. Isaksen, and Kenneth Lauer This document is a working paper, the purposes of which are to describe the three

More information

Lecture 1: Machine Learning Basics

Lecture 1: Machine Learning Basics 1/69 Lecture 1: Machine Learning Basics Ali Harakeh University of Waterloo WAVE Lab ali.harakeh@uwaterloo.ca May 1, 2017 2/69 Overview 1 Learning Algorithms 2 Capacity, Overfitting, and Underfitting 3

More information

Getting Started with Deliberate Practice

Getting Started with Deliberate Practice Getting Started with Deliberate Practice Most of the implementation guides so far in Learning on Steroids have focused on conceptual skills. Things like being able to form mental images, remembering facts

More information

Implementing a tool to Support KAOS-Beta Process Model Using EPF

Implementing a tool to Support KAOS-Beta Process Model Using EPF Implementing a tool to Support KAOS-Beta Process Model Using EPF Malihe Tabatabaie Malihe.Tabatabaie@cs.york.ac.uk Department of Computer Science The University of York United Kingdom Eclipse Process Framework

More information

What is beautiful is useful visual appeal and expected information quality

What is beautiful is useful visual appeal and expected information quality What is beautiful is useful visual appeal and expected information quality Thea van der Geest University of Twente T.m.vandergeest@utwente.nl Raymond van Dongelen Noordelijke Hogeschool Leeuwarden Dongelen@nhl.nl

More information

Universal Design for Learning Lesson Plan

Universal Design for Learning Lesson Plan Universal Design for Learning Lesson Plan Teacher(s): Alexandra Romano Date: April 9 th, 2014 Subject: English Language Arts NYS Common Core Standard: RL.5 Reading Standards for Literature Cluster Key

More information

Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge

Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Innov High Educ (2009) 34:93 103 DOI 10.1007/s10755-009-9095-2 Maximizing Learning Through Course Alignment and Experience with Different Types of Knowledge Phyllis Blumberg Published online: 3 February

More information

OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE

OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE Mark R. Shinn, Ph.D. Michelle M. Shinn, Ph.D. Formative Evaluation to Inform Teaching Summative Assessment: Culmination measure. Mastery

More information

Practice Examination IREB

Practice Examination IREB IREB Examination Requirements Engineering Advanced Level Elicitation and Consolidation Practice Examination Questionnaire: Set_EN_2013_Public_1.2 Syllabus: Version 1.0 Passed Failed Total number of points

More information

What Is a Chief Diversity Officer? By. Dr. Damon A. Williams & Dr. Katrina C. Wade-Golden

What Is a Chief Diversity Officer? By. Dr. Damon A. Williams & Dr. Katrina C. Wade-Golden What Is a Chief Diversity Officer? By Dr. Damon A. Williams & Dr. Katrina C. Wade-Golden To meet the needs of increasingly diverse campuses, many institutions have developed executive positions to guide

More information

Australia s tertiary education sector

Australia s tertiary education sector Australia s tertiary education sector TOM KARMEL NHI NGUYEN NATIONAL CENTRE FOR VOCATIONAL EDUCATION RESEARCH Paper presented to the Centre for the Economics of Education and Training 7 th National Conference

More information

Educational Attainment

Educational Attainment A Demographic and Socio-Economic Profile of Allen County, Indiana based on the 2010 Census and the American Community Survey Educational Attainment A Review of Census Data Related to the Educational Attainment

More information

Cooking Matters at the Store Evaluation: Executive Summary

Cooking Matters at the Store Evaluation: Executive Summary Cooking Matters at the Store Evaluation: Executive Summary Introduction Share Our Strength is a national nonprofit with the goal of ending childhood hunger in America by connecting children with the nutritious

More information

Game-based formative assessment: Newton s Playground. Valerie Shute, Matthew Ventura, & Yoon Jeon Kim (Florida State University), NCME, April 30, 2013

Game-based formative assessment: Newton s Playground. Valerie Shute, Matthew Ventura, & Yoon Jeon Kim (Florida State University), NCME, April 30, 2013 Game-based formative assessment: Newton s Playground Valerie Shute, Matthew Ventura, & Yoon Jeon Kim (Florida State University), NCME, April 30, 2013 Fun & Games Assessment Needs Game-based stealth assessment

More information

STANDARDS AND RUBRICS FOR SCHOOL IMPROVEMENT 2005 REVISED EDITION

STANDARDS AND RUBRICS FOR SCHOOL IMPROVEMENT 2005 REVISED EDITION Arizona Department of Education Tom Horne, Superintendent of Public Instruction STANDARDS AND RUBRICS FOR SCHOOL IMPROVEMENT 5 REVISED EDITION Arizona Department of Education School Effectiveness Division

More information

Loughton School s curriculum evening. 28 th February 2017

Loughton School s curriculum evening. 28 th February 2017 Loughton School s curriculum evening 28 th February 2017 Aims of this session Share our approach to teaching writing, reading, SPaG and maths. Share resources, ideas and strategies to support children's

More information

Essay on importance of good friends. It can cause flooding of the countries or even continents..

Essay on importance of good friends. It can cause flooding of the countries or even continents.. Essay on importance of good friends. It can cause flooding of the countries or even continents.. Essay on importance of good friends >>>CLICK HERE

More information

Listening to your members: The member satisfaction survey. Presenter: Mary Beth Watt. Outline

Listening to your members: The member satisfaction survey. Presenter: Mary Beth Watt. Outline Listening to your members: The satisfaction survey Listening to your members: The member satisfaction survey Presenter: Mary Beth Watt 1 Outline Introductions Members as customers Member satisfaction survey

More information

SASKATCHEWAN MINISTRY OF ADVANCED EDUCATION

SASKATCHEWAN MINISTRY OF ADVANCED EDUCATION SASKATCHEWAN MINISTRY OF ADVANCED EDUCATION Report March 2017 Report compiled by Insightrix Research Inc. 1 3223 Millar Ave. Saskatoon, Saskatchewan T: 1-866-888-5640 F: 1-306-384-5655 Table of Contents

More information

The Good Judgment Project: A large scale test of different methods of combining expert predictions

The Good Judgment Project: A large scale test of different methods of combining expert predictions The Good Judgment Project: A large scale test of different methods of combining expert predictions Lyle Ungar, Barb Mellors, Jon Baron, Phil Tetlock, Jaime Ramos, Sam Swift The University of Pennsylvania

More information

Reducing Spoon-Feeding to Promote Independent Thinking

Reducing Spoon-Feeding to Promote Independent Thinking Reducing Spoon-Feeding to Promote Independent Thinking Janice T. Blane This paper was completed and submitted in partial fulfillment of the Master Teacher Program, a 2-year faculty professional development

More information

The Effect of Income on Educational Attainment: Evidence from State Earned Income Tax Credit Expansions

The Effect of Income on Educational Attainment: Evidence from State Earned Income Tax Credit Expansions The Effect of Income on Educational Attainment: Evidence from State Earned Income Tax Credit Expansions Katherine Michelmore Policy Analysis and Management Cornell University km459@cornell.edu September

More information

others have examples for how feedback mechanisms at the CBO level have been established?

others have examples for how feedback mechanisms at the CBO level have been established? Dear Data Use Net members, Thanks for the many contributions for Day 3 we appreciate the time you took to submit your comments and examples. As a reminder for other readers, while we ask for your name

More information

CHAPTER 2: COUNTERING FOUR RISKY ASSUMPTIONS

CHAPTER 2: COUNTERING FOUR RISKY ASSUMPTIONS CHAPTER 2: COUNTERING FOUR RISKY ASSUMPTIONS PRESENTED BY GAMES FOR CHANGE AND THE MICHAEL COHEN GROUP FUNDED BY THE DAVID & LUCILE PACKARD FOUNDATION ADVISORY BOARD CHAIR: BENJAMIN STOKES, PHD Project

More information

Co-teaching in the ESL Classroom

Co-teaching in the ESL Classroom WINTER 2008 Co-teaching in the ESL Classroom BY ANDREA HONIGSFELD AND MARiA DOVE The authors explore the transferability of coteaching models and techniques from the field of Special Education to that

More information

Stakeholder Engagement and Communication Plan (SECP)

Stakeholder Engagement and Communication Plan (SECP) Stakeholder Engagement and Communication Plan (SECP) Summary box REVIEW TITLE 3ie GRANT CODE AUTHORS (specify review team members who have completed this form) FOCAL POINT (specify primary contact for

More information

Writing Research Articles

Writing Research Articles Marek J. Druzdzel with minor additions from Peter Brusilovsky University of Pittsburgh School of Information Sciences and Intelligent Systems Program marek@sis.pitt.edu http://www.pitt.edu/~druzdzel Overview

More information

Best Practices in Internet Ministry Released November 7, 2008

Best Practices in Internet Ministry Released November 7, 2008 Best Practices in Internet Ministry Released November 7, 2008 David T. Bourgeois, Ph.D. Associate Professor of Information Systems Crowell School of Business Biola University Best Practices in Internet

More information

THINKING TOOLS: Differentiating the Content. Nanci Cole, Michelle Wikle, and Sacha Bennett - TOSAs Sandi Ishii, Supervisor of Gifted Education

THINKING TOOLS: Differentiating the Content. Nanci Cole, Michelle Wikle, and Sacha Bennett - TOSAs Sandi Ishii, Supervisor of Gifted Education THINKING TOOLS: Differentiating the Content Nanci Cole, Michelle Wikle, and Sacha Bennett - TOSAs Sandi Ishii, Supervisor of Gifted Education Based on training by: S. Kaplan, USC, 2008 What is Academic

More information

Five Challenges for the Collaborative Classroom and How to Solve Them

Five Challenges for the Collaborative Classroom and How to Solve Them An white paper sponsored by ELMO Five Challenges for the Collaborative Classroom and How to Solve Them CONTENTS 2 Why Create a Collaborative Classroom? 3 Key Challenges to Digital Collaboration 5 How Huddle

More information

Karla Brooks Baehr, Ed.D. Senior Advisor and Consultant The District Management Council

Karla Brooks Baehr, Ed.D. Senior Advisor and Consultant The District Management Council Karla Brooks Baehr, Ed.D. Senior Advisor and Consultant The District Management Council This paper aims to inform the debate about how best to incorporate student learning into teacher evaluation systems

More information

Faculty Schedule Preference Survey Results

Faculty Schedule Preference Survey Results Faculty Schedule Preference Survey Results Surveys were distributed to all 199 faculty mailboxes with information about moving to a 16 week calendar followed by asking their calendar schedule. Objective

More information

CHAPTER 4: REIMBURSEMENT STRATEGIES 24

CHAPTER 4: REIMBURSEMENT STRATEGIES 24 CHAPTER 4: REIMBURSEMENT STRATEGIES 24 INTRODUCTION Once state level policymakers have decided to implement and pay for CSR, one issue they face is simply how to calculate the reimbursements to districts

More information

OFFICE OF ENROLLMENT MANAGEMENT. Annual Report

OFFICE OF ENROLLMENT MANAGEMENT. Annual Report 2014-2015 OFFICE OF ENROLLMENT MANAGEMENT Annual Report Table of Contents 2014 2015 MESSAGE FROM THE VICE PROVOST A YEAR OF RECORDS 3 Undergraduate Enrollment 6 First-Year Students MOVING FORWARD THROUGH

More information

Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses

Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses Designing a Rubric to Assess the Modelling Phase of Student Design Projects in Upper Year Engineering Courses Thomas F.C. Woodhall Masters Candidate in Civil Engineering Queen s University at Kingston,

More information

2 Participatory Learning and Action Research (PLAR) curriculum

2 Participatory Learning and Action Research (PLAR) curriculum 2 Participatory Learning and Action Research (PLAR) curriculum 2.1 Principles and objectives of the PLAR approach approach, based on adult learning in groups of 20 to 25 farmers, curriculum covers the

More information

Higher Education Review (Embedded Colleges) of Navitas UK Holdings Ltd. Hertfordshire International College

Higher Education Review (Embedded Colleges) of Navitas UK Holdings Ltd. Hertfordshire International College Higher Education Review (Embedded Colleges) of Navitas UK Holdings Ltd April 2016 Contents About this review... 1 Key findings... 2 QAA's judgements about... 2 Good practice... 2 Theme: Digital Literacies...

More information

Calculators in a Middle School Mathematics Classroom: Helpful or Harmful?

Calculators in a Middle School Mathematics Classroom: Helpful or Harmful? University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Action Research Projects Math in the Middle Institute Partnership 7-2008 Calculators in a Middle School Mathematics Classroom:

More information

Virtual Seminar Courses: Issues from here to there

Virtual Seminar Courses: Issues from here to there 1 of 5 Virtual Seminar Courses: Issues from here to there by Sherry Markel, Ph.D. Northern Arizona University Abstract: This article is a brief examination of some of the benefits and concerns of virtual

More information

Davidson College Library Strategic Plan

Davidson College Library Strategic Plan Davidson College Library Strategic Plan 2016-2020 1 Introduction The Davidson College Library s Statement of Purpose (Appendix A) identifies three broad categories by which the library - the staff, the

More information

Case study Norway case 1

Case study Norway case 1 Case study Norway case 1 School : B (primary school) Theme: Science microorganisms Dates of lessons: March 26-27 th 2015 Age of students: 10-11 (grade 5) Data sources: Pre- and post-interview with 1 teacher

More information

How to make your research useful and trustworthy the three U s and the CRITIC

How to make your research useful and trustworthy the three U s and the CRITIC How to make your research useful and trustworthy the three U s and the CRITIC Michael Wood University of Portsmouth Business School http://woodm.myweb.port.ac.uk/sl/researchmethods.htm August 2015 Introduction...

More information