SMALL GROUP BRAINSTORMING AND IDEA QUALITY Is Electronic Brainstorming the Most Effective Approach?

SMALL Barki, Pinsonneault GROUP RESEARCH / BRAINSTORMING / April 2001AND IDEA QUALITY SMALL GROUP BRAINSTORMING AND IDEA QUALITY Is Electronic Brainstorming the Most Effective Approach? HENRI BARKI École des Hautes Études Commerciales ALAIN PINSONNEAULT McGill University The importance of idea quality as an indicator of brainstorming groups performance, evidence showing that idea quantity may not be a good substitute for idea quality, that most electronic brainstorming (EBS) research has operationalized group performance with idea quantity, and that little direct empirical evidence exists comparing the quality of ideas produced by EBS and nominal brainstorming groups suggest the need for a closer look at EBS and nominal brainstorming group performance in terms of idea quality. This study employed an unbalanced repeated measures experimental design to compare the quality of ideas generated by small groups using four brainstorming technologies (nominal, verbal, EBS-anonymous, and EBS-non-anonymous). In the experiment, three conditions thought to improve the effectiveness of EBS groups were also manipulated. The results indicated that overall, nominal brainstorming groups generated ideas at least as good, if not better, than EBS groups. Furthermore, the three conditions manipulated did not improve the quality of the ideas generated by EBS groups. To survive in today s complex, dynamic, and highly competitive environment, organizations need to constantly innovate, a capacity that rests largely on the ability of employees to be creative and to generate novel ideas of high quality (Amabile, 1997; Herrmann, AUTHORS NOTE: Financial support for this research was provided by the Social Sciences and Humanities Research Council of Canada and the Fonds pour la Formation Chercheurs et l Aide à la Recherche of Québec. Thanks are extended to R. Brent Gallupe and Norberto Hoppen for their help and advice in conducting the experiment. SMALL GROUP RESEARCH, Vol. 32 No. 2, April 2001 158-205 2001 Sage Publications, Inc. 158

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 159 1993; Massetti, 1996). Brainstorming (or verbal brainstorming), initially proposed by Osborn (1957), is an approach designed to help the generation of new ideas by asking members of a group to freely generate as many ideas about a topic as they can and verbally express them one at a time. Subsequent to its early popularity, the effectiveness of this approach has often been questioned (Stroebe & Diehl, 1994). Indeed, evidence from numerous studies in social psychology and group psychology has shown that groups generate better ideas with nominal brainstorming, where group members generate ideas individually without communicating with other members of the group, than with verbal brainstorming (Diehl & Stroebe, 1987, 1991; Gurman, 1968; Mullen, Johnson, & Salas, 1991; Stroebe & Diehl, 1994). Electronic brainstorming (EBS) has been proposed as an approach that overcomes the shortcomings of verbal brainstorming (e.g., production blocking, evaluation apprehension, free riding) while stimulating the production of good ideas. Because they work simultaneously, as well as generating and sharing their ideas anonymously, members of EBS groups are expected to generate ideas of higher quality than verbal or nominal brainstorming groups (Nunamaker, Dennis, Valacich, Vogel, & George, 1991). However, whereas empirical evidence clearly shows EBS to be superior to verbal brainstorming, the same cannot be said for EBS versus nominal brainstorming. In fact, both studies that compared the quality of ideas generated with EBS and nominal brainstorming groups (Dennis & Valacich, 1994; Valacich, Dennis, & Connolly, 1994) found no differences between the two techniques for small groups. Examining the performance of EBS and nominal brainstorming groups in terms of the quality of the ideas they generate is important because a key purpose of idea generation is the identification of a few good or interesting ideas with a view to implementing one of them. People value quality over quantity as a goal of brainstorming (Rowatt, Nesselroade, Beggan, & Allison, 1997). Yet, in part backed by the premise that quantity breeds quality (Osborn, 1957), most EBS idea-generation research has measured group performance with idea quantity (Nagasundaram & Bostrom, 1994-1995). How-

160 SMALL GROUP RESEARCH / April 2001 ever, empirical support for this premise is mixed. For example, Gryskiewicz (1988) found that idea quality was relatively independent of idea quantity and that although brainwriting 1 produced the highest number of ideas, nominal brainstorming produced ideas that were more novel, effective, and creative. In addition, groups using the excursion technique 2 produced fewer ideas, but they generated the most useful ideas as judged by experts. Similarly, Buyer (1978) found that although brainstorming instructions lead to more ideas, criteria-cues instructions 3 lead to ideas of higher quality (as measured by the percentage of superior responses and mean creativity). Buyer also found no differences between the two approaches in terms of the number of superior responses and the creativity rating of the best and preferred responses. Thus, the importance of idea quality as an indicator of brainstorming groups performance, evidence showing that idea quantity may not be a good substitute for idea quality, that most EBS research operationalized group performance with idea quantity, and that little direct empirical evidence exists comparing the quality of ideas produced by EBS and nominal brainstorming groups suggest the need for a closer look at EBS and nominal brainstorming group performance in terms of idea quality. This article addresses this issue by comparing the quality of ideas generated by EBS and nominal brainstorming groups and by examining whether EBS group performance can be improved. 4 To do so, the effect of three conditions thought to improve EBS group performance are examined using small groups. 5 The article is organized as follows. First, the literature and the existing empirical evidence on idea quality and four brainstorming approaches (verbal, nominal, EBS-anonymous, and EBS-non-anonymous) are reviewed. Second, the process gains and losses of each brainstorming approach are assessed, and their ability to generate high quality ideas are compared. Subsequently, three factors expected to improve EBS s performance are introduced, and their potential impacts are analyzed. Next, the study s research hypotheses, the experiment conducted for testing them, and the results of the study are presented. This is followed by a discussion of the results, their implications for future research, and the study s conclusions.

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 161 BRAINSTORMING AND IDEA QUALITY Nominal brainstorming has consistently been found to produce higher quality ideas than verbal brainstorming. In a review of studies that compared verbal and nominal brainstorming, Diehl and Stroebe (1987) noted that nominal brainstorming groups produced ideas of higher total quality than verbal brainstorming groups, in all six studies that were reviewed (Bouchard, 1969; Dunnette, Campbell, & Jaastad, 1963; Gurman, 1968; Milton, 1965; Taylor, Berry, & Block, 1958; Vroom, Grant, & Cotton, 1969), and higher number of good ideas in two of three studies (Bouchard, 1969; Vroom et al., 1969). Diehl and Stroebe s (1987, 1991) results also indicated that nominal brainstorming groups produced more good ideas than verbal brainstorming groups. The results for average idea quality were equivocal in all four studies (Bouchard, 1969; Dunnette et al., 1963; Taylor et al., 1958; Vroom et al., 1969). It is argued that by allowing participants to see other participants ideas on their screen, EBS promotes group synergy and stimulation (Dennis, Hayes, & Daniels, 1999; Satzinger, Garfield, & Nagasundaram, 1999). This is thought to facilitate the construction of chains of thought, to enable participants to build on good ideas generated by other group members, and to encourage them to think in novel directions, thus spurring new ideas and improving creativity and originality. However, there is little empirical evidence supporting such a stimulation effect. In fact, precise cues such as those provided by EBS have been found to have little effect on idea quality (Sobel & Rothenberg, 1980). Moreover, collaborative technologies have not been found to improve group output quality (Horton, Rogers, Austin, & McCormick, 1991-1992). Empirical evidence comparing the quality of ideas generated by small EBS and nominal brainstorming groups is also mixed. A recent review by Fjermestad and Hiltz (1998-1999) identified 184 EBS studies, of which 3 compared the quality of ideas generated by EBS and verbal brainstorming groups (Gallupe et al., 1992; Petrovic & Krickl, 1994; Valacich, Paranka, George, & Nunamaker, 1993). All 3 studies found EBS groups to generate higher quality ideas than verbal groups. Eight other studies also assessed idea quality, but

162 SMALL GROUP RESEARCH / April 2001 they compared groups using EBS with other groups using EBS, under different experimental conditions (Connolly, Jessup, & Valacich, 1990; Dennis, Aronson, Heninger, & Walker, 1999; Dennis, Valacich, Connolly, & Wynne, 1996; Dennis, Valacich, & Nunamaker, 1990; Valacich, Dennis, Nunamaker, 1992; Valacich, George, Nunamaker, & Vogel, 1994; Valacich, Mennecke, Wachter, & Wheeler, 1993; Valacich, Wheeler, Mennecke, & Wachter, 1995). Only two studies compared the quality of ideas generated by EBS and nominal brainstorming groups (Dennis & Valacich, 1994; Valacich, Dennis, & Connolly, 1994). Dennis and Valacich (1994) reported the results of two sets of analyses. In the first set, 18- member groups were found to have higher total quality scores than nominal groups, with no differences found for 9-member, 3- member, and nominal groups. In the second set, 12-member groups were found to have higher total quality scores than nominal groups, while nominal groups were found to have higher total quality scores than 4-member groups. In their other study, Valacich, Dennis, and Connolly (1994) reported two experiments where the quality of ideas generated by EBS and nominal groups were compared. In the first experiment, no differences were found in average idea quality between EBS and nominal brainstorming groups of size 3, 9, and 18, while the total quality scores of 18-member EBS groups were found to be greater than the equivalent nominal groups. In the second experiment, no differences were found in average idea quality between EBS and nominal brainstorming groups of size 4, 8, and 12, while the total quality scores of 12-member EBS groups were found to be greater than the equivalent nominal groups. The results of these two studies, although relatively supportive of EBS s superiority over nominal brainstorming for large groups, are not conclusive and need to be replicated with measures of idea quality different from total quality (which tends to be highly correlated with idea quantity). In summary, then, the theoretical explanations that suggest that EBS groups ought to generate ideas of higher quality than nominal brainstorming groups have received mixed empirical support, and existing empirical evidence that compares the quality of ideas generated by EBS and nominal brainstorming groups is not clear.

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 163 BRAINSTORMING AND IDEA QUALITY: A PROCESS GAINS AND PROCESS LOSSES ASSESSMENT Applying the framework suggested by Pinsonneault, et al. (1999a) to idea quality, Table 1 compares the process losses and gains (a process loss refers to a phenomenon negatively affecting a brainstorming group s effectiveness, while a process gain refers to a positive impact) inherent to EBS, nominal, and verbal brainstorming, and it provides an estimate of the effectiveness of each brainstorming technology by summing its process gains and losses. Consistent with the theoretical arguments and empirical evidence reviewed previously, this analysis suggests verbal brainstorming to have the lowest potential for generating high quality ideas (5 gains and 10 losses, giving a net score of 5), while nominal, EBS-anonymous, and EBS-non-anonymous are roughly equivalent (with respective net scores of 1, 1, 1). However, before concluding that EBS is not superior to nominal brainstorming, an additional question needs to be answered: Are there conditions that help improve the quality of ideas generated by EBS groups, making them more productive than nominal groups? Three such conditions or factors can be gleaned from the literature: Group History, Social Sensitivity of the idea generation topic, and Contextual Cues. Table 2 presents how these factors can be expected to affect the different process gains and process losses identified in Table 1 for verbal, nominal, EBS-anonymous, and EBS-non-anonymous brainstorming technologies. In Table 2, increased process gains and process losses due to a given factor are indicated by a ++ and a, respectively. The symbols used to represent the three factors are [ ] for Group History, { } for Topic Sensitivity, and ( ) for Contextual Cues. For example, vague contextual cues (an experimental factor explained below) are expected to reinforce a process gain (cognitive stimulation) inherent to EBS-anonymous. This is illustrated by (++) in Table 2 on the cognitive stimulation row. When a factor introduces a new process gain or process loss, it is indicated by a single sign within a symbol representing the factor. For example, vague con- Text continues on p. 170

164 TABLE 1: Process Gains and Process Losses of Four Brainstorming Technologies Brainstorming Technology Verbal Nominal EBS- EBS-Non- Anonymous Anonymous Process gains Procedural mechanisms (characteristics of the group process used for generating ideas; Camacho & Paulus, 1995; Diehl & Stroebe, 1987; Mullen, Johnson, & Salas, 1991) Separation of task processes Separation of ideation and evaluation increase + + + quality (Brilhart & Jochem, 1964). Social psychological mechanisms (social and psychological effects associated with the presence of other people and by individual membership in a group; Camacho & Paulus, 1995; Diehl & Stroebe, 1987; Mullen et al., 1991) Cognitive stimulation or Good ideas spur more good ideas, and member + + + synergy utterances may contain task-related stimuli that elicit new ideas from other members (Dennis & Valacich, 1993; Lamm & Trommsdorf, 1973; Osborn, 1957; Straus, 1996).

Observational modeling Members can learn from and imitate best perform- + + (or observational learning) ers hence increasing the quality of group output (Amabile, 1996; Hill, 1982; Lamm & Trommsdorf, 1973). Divergent thinking (lateral Members ability to look for new and innovative + + + thinking, breath of attention) ideas results in better ideas. Divergent thinking can be improved by techniques allowing to approach a problem from different vantage points (Adam, 1979; Basadur, Wakabayashi, & Graen, 1990; DeBono, 1971; Farr, 1990; Guilford, 1967, 1977; Kasof, 1997; Woodman, Sawyer, & Griffin, 1993). Self-presentation and social Presence of others causes people to perform simple + + recognition tasks better (i.e., generating good ideas), and individuals want their contributions to be recognized by others (Bond, 1982; Bond & Titus, 1983; Goethals & Darley, 1987; Mullen & Goethals, 1987). Task orientation Idea quality is improved when discussions are task- + + + oriented rather than socializing (McGrath, 1984; Nunamaker et al., 1991; Shaw, 1981). Economic mechanisms (factors that affect the motivation of individuals and their desire to contribute to group outcomes; Camacho & Paulus, 1995; Diehl & Stroebe, 1987; Mullen et al., 1991) 165 (continued)

166 TABLE 1: Continued Brainstorming Technology Verbal Nominal EBS- EBS-Non- Anonymous Anonymous Motivation or arousal Working in groups stimulates individuals to per- + + + + form better (Lamm & Trommsdorf, 1973, Shaw, 1981; Zajonc, 1965). Total process gains 5 3 5 7 Process losses Procedural mechanisms Production blocking Being unable to express good ideas as they occur impairs performance because (a) ideas become irrelevant, (b) people forget their ideas, (c) to avoid forgetting ideas, people rehearse ideas, which prevents them from generating new good ideas (Diehl & Stroebe, 1987, 1991; Lamm & Trommsdorf, 1973; Paulus, Brown, & Ortega, 1997; Stroebe & Diehl, 1994). Effort redundancy Generating duplicate good ideas (Lamm & Trommsdorf, 1973). Cognitive interference Content of ideas generated by other members might interfere with one s internal idea generation (e.g., being unable to generate new good ideas because one is entranced by other group members ideas) (Lamm & Trommsdorf, 1973; Straus, 1996).

Cognitive inertia (convergent Individuals might embark on a single train of Thinking) thought, which limits creativity and the generation of good ideas (DeBono, 1968, 1971; Jablin & Seibold, 1978; Lamm & Trommsdorf, 1973). Social psychological mechanisms Evaluation apprehension When members fear expressing ideas because of potential retaliation, creativity and the production of good ideas are impaired (Camacho & Paulus, 1995; Diehl & Stroebe, 1987; Jablin & Seibold, 1978; Lamm & Trommsdorf, 1973; Mullen et al., 1991) Negative quality matching Comparison and adjustment of individual perform- ance to a baseline idea quality level (Goethals & Darley, 1987; Paulus & Dzindolet, 1993; Paulus et al., 1997; Seta, Seta, & Donaldson, 1991). Pressure for cognitive uni- Members may feel pressure to remain within group formity or conformity or social norms (Hackman & Kaplan, 1974; Hoffman, 1965; Janis, 1972; McGrath, 1984). Personalization of issues Members associating the discussion of issues to personal matters (McGrath, 1984). Social monitoring Because other members, by their presence, create uncertainty and a readiness to respond, the individual must periodically observe them in an attempt to anticipate what they may do, thus hindering their creativity (Mullen & Goethals, 1987). (continued) 167

168 TABLE 1: Continued Brainstorming Technology Verbal Nominal EBS- EBS-Non- Anonymous Anonymous Social inhibition Presence of others may disrupt an individual s creative performance and ability to generate good ideas when others are concentrating on the individual s work (Amabile, 1996). Social influence Domination by a few members exercising undue influence can limit creativity and the production of good ideas (Jablin & Seibold, 1978; McGrath, 1984; Nunamaker et al., 1991). Economic mechanisms Free riding (social loafing) Members might intentionally limit their efforts and contributions by relying on others to generate high quality ideas because of (a) perceived dispensability of one s effort (Harkins & Petty, 1982; Kerr & Bruun, 1983), (b) diffused responsibility (Latané, Williams, & Harkins, 1979; Williams, Harkins, & Latané, 1981), or (c) social and cognitive loafing (Albanese & Van Fleet, 1985; Diehl & Stroebe, 1987; Mullen & Goethals, 1987). Total process losses 10 2 4 8 Net score (gains losses) 5 1 1 1 NOTE: + = process gain present, - = process loss present.

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 169 TABLE 2: Process Gains, Process Losses, and Study Factors Brainstorming Technology EBS- EBS-Non- Verbal Nominal Anonymous Anonymous Process gains Procedural mechanisms Separation of task processes + + + Social psychological mechanisms Cognitive stimulation or synergy (++) (+) (++) (++) Observational modeling (observational learning) [++] [++] Divergent thinking (lateral thinking, breath of attention) (++) (++) (++) Self-presentation and social recognition [++] [++] Task orientation + [++] [++] Economic mechanisms Motivation or arousal {++} {++} {++} {++} Total gains [established] 7 2 6 10 Total gains {sensitive topics} 6 4 6 8 Total gains (contextual cues) 7 3 7 9 Process losses Procedural mechanisms Production blocking Effort redundancy Cognitive interference ( ) ( ) ( ) Cognitive inertia (convergent thinking) ( ) Social psychological mechanisms Evaluation apprehension { } { } Negative quality matching { } { } { } Pressure for cognitive uniformity/ conformity { } { } Personalization of issues Social monitoring Social inhibition Social influence Economic mechanisms Free riding (social loafing) [ ] [ ] Total losses [established] 9 2 3 8 Total losses {sensitive topics} 11 2 3 9 Total losses (contextual cues) 9 1 3 7 Net score [established] 2 0 3 2 Net score {sensitive topics} 5 2 3 1 Net score (contextual cues) 2 2 4 2 (continued)

170 SMALL GROUP RESEARCH / April 2001 TABLE 2: Continued Brainstorming Technology EBS- EBS-Non- Verbal Nominal Anonymous Anonymous Overall net score 9 0 10 3 NOTE: + = Gain present in a group using a particular technology (see Table 1) and not affected by any factor. = loss present in a group using a particular technology (see Table 1) and not affected by any factor. ++ = Gain reinforced by a factor (present without the factor (see Table 1) and strengthened by a factor). = loss reinforced by a factor (present without the factor (see Table 1) and strengthened by a factor). (-) = loss present with vague contextual cues [-] = loss present with established groups {-} = loss present with sensitive topics (+) = Gain present with vague contextual cues [+] = Gain present with established groups {+} = Gain present with sensitive topics ( ) = loss or gain eliminated with vague contextual cues [ ] = loss or gain eliminated with established groups { } = loss or gain eliminated with sensitive topics. The counts are calculated as follows. For example, for established groups, verbal technology has 6 process gains (1 cognitive stimulation, 2 observational learning, 2 Social recognition, 1 motivation or arousal). Similarly, when vague contextual cues are present (seeded condition), verbal brainstorming has 5 gains (2 cognitive stimulation, 1 observational learning, 1 social recognition, 1 motivation or arousal). Process losses are calculated in a similar fashion. For example, nominal has 1 process loss when vague contextual cues are present (1 Effort redundancy due to cognitive inertia being eliminated by the condition), but it has 2 losses in the case of established groups (1 effort redundancy and 1 cognitive inertia, which in this case, is not eliminated by the factor). EBS = electronic brainstorming. textual cues are expected to introduce a new process gain (i.e., cognitive stimulation) to nominal brainstorming. This is represented by (+) in Table 2. A process loss eliminated by a condition is illustrated in Table 2 by an empty symbol. For example, it is assumed that when established groups brainstorm, the free-riding process loss will be eliminated with verbal and EBS-anonymous technologies, hence the empty symbol [ ] in Table 2. Finally, a process gain or process loss not affected by any of the three factors is simply represented in Table 2 by a + or a, as it was in Table 1. Table 2 also shows the net score of each brainstorming technology under each condition (see Table 2 for an example).

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 171 BRAINSTORMING AND IDEA QUALITY: THREE FACTORS THAT MAY AFFECT EBS S EFFECTIVENESS Group History refers to the extent to which a group has previously worked together. Most research on EBS has been conducted with ad hoc groups typically made up of individuals randomly brought together for the purposes of a study. Established groups, on the other hand, have a meaningful history of working together, with established norms of group behavior and performance, and are used to working together, with most of their efforts in a group session oriented toward resolving the problem at hand (Tuckman, 1965; Wheelan & Hochberger, 1996). In addition, because they share a common interpretative context (or transactive memory), established group members communicate more effectively and understand each other better, even with a less rich media (Mullen & Goethals, 1987; Zack, 1993). Such groups also tend to follow a variety of paths during decision making and were found to be more creative in their interaction processes (Poole & Baldwin, 1996). In contrast, groups having no history of having previously worked together may not be able to take full advantage of EBS capabilities. Established groups working together should be better able to benefit from the stimulation-synergy-creativity gains than when their members work alone (i.e., nominal brainstorming). Hence, established EBS groups may generate ideas of higher quality than both established and ad hoc nominal brainstorming groups. Consistent with this reasoning, Table 2 shows that for established groups, EBS-anonymous and EBS-non-anonymous have net scores of 3 and 2, respectively, whereas verbal and nominal brainstorming obtain scores of 2 and 0, respectively. Topic Sensitivity refers to the degree to which a brainstorming topic is socially sensitive or controversial. Group performance can be diminished when participants do not find the brainstorming topic to be sufficiently stimulating (Fiske & Taylor, 1991; Karau & Williams, 1993; Pinsonneault & Heppel, 1998). Socially sensitive topics such as how to prevent the spread of AIDS or violent crime can be expected to generate greater interest, controversy, and excitement than socially less sensitive topics, providing greater stimulation to a brainstorming group. Furthermore, brainstorming

172 SMALL GROUP RESEARCH / April 2001 on a socially sensitive topic under anonymous conditions may decrease feelings of evaluation apprehension, allowing individuals to express original ideas they may not have otherwise mentioned. Thus, groups can be expected to generate ideas of higher quality when brainstorming anonymously on socially sensitive topics than without anonymity. Consequently, to the extent that the stimulation provided by EBS interacts with the added stimulation provided by a socially sensitive topic, it could result in EBS-anonymous groups generating better ideas than nominal brainstorming groups. Consistent with this, the process losses or gains assessment of Table 2 indicates that for sensitive topics, EBS-anonymous has a net score of 3, whereas nominal brainstorming, EBS-non-anonymous, and verbal brainstorming have net scores of 2, 1, and 5, respectively. A third condition under which EBS groups could produce ideas of higher quality than nominal brainstorming is the presence of contextual cues during a brainstorming session. These refer to supplemental environmental stimuli that provide hints or clues that help participants in performing the task at hand (Amabile, 1983; Paulus & Dzindolet, 1993). Although idea sharing provides cues, such cues may not be stimulating enough for spurring new, original, and high-quality ideas (Sobel & Rothenberg, 1980). Research in social psychology and group psychology indicates that the stimulation effect of environments rich in precise cues (where simple pictures or other objects provide precise clues to responses) is very limited (Friedman, Raymond, & Feldhusen, 1978; Higgins & Chaires, 1980; La Greca, 1980; Paulus & Dzindolet, 1993; Paulus, Dzindolet, Poletes, & Gamacho, 1993; Sobel & Rothenberg, 1980; Ward, 1969). On the other hand, environments rich in vague cues (e.g., where superimposed pictures or other objects provide imprecise clues to possible responses) were found to foster cognitive stimulation and improve creativity (Amabile, 1983). In addition, breath of attention (the number and range of environmental stimuli attended to at any one time) was found to positively correlate with creativity and quality (Kasof, 1997). Consequently, providing additional vague cues (i.e., ideas of high quality along different dimensions of a particular topic) might raise the stimulation felt by EBS groups, who would now face two sources of stimulation: contextual

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 173 cues and other participants ideas that they can read on their screens. If so, the stimulation provided by vague contextual cues combined with the stimulation inherent to EBS could allow EBS groups to generate better ideas than nominal brainstorming groups (who would have only the contextual cues as a stimulation source). The assessment of the four brainstorming technologies (in Table 2) indicates that under the contextual cues condition, EBS-anonymous could surpass nominal and EBS-non-anonymous (net scores of 4, 2, and 2, respectively), with all three expected to surpass verbal brainstorming (net score of 2). Table 2 also indicates that overall, when all three EBS-favoring conditions are present (i.e., established groups generating ideas on a sensitive topic in the presence of contextual cues), EBS-anonymous groups (net score = 10) could generate ideas of higher quality than EBS-non-anonymous (net score = 3), nominal brainstorming (net score = 0), and verbal brainstorming groups (net score = 9). The above discussion can be summarized as follows. Although EBS is thought to foster the production of better ideas than other brainstorming technologies in the IS literature, neither the existing conceptual arguments nor the available empirical evidence appear to support this view. The process losses / process gains analysis presented in Table 1 converges with this conclusion, indicating that in the absence of facilitating conditions, there should be no difference in the quality of ideas produced by EBS and nominal groups, hence our primary hypothesis: Hypothesis 1: There will be no difference in the quality of ideas generated between EBS groups and nominal brainstorming groups. However, the possibility exists that the three factors discussed above and assessed in Table 2 could improve the quality of ideas generated by EBS groups over nominal brainstorming, hence the following hypotheses: Hypothesis 2: Established groups using EBS will generate ideas of higher quality than established groups using nominal brainstorming. Hypothesis 3: When Topic Sensitivity is high, EBS groups will generate higher quality ideas than nominal brainstorming groups.

174 SMALL GROUP RESEARCH / April 2001 Hypothesis 4: In the presence of vague contextual cues, EBS groups will generate higher quality ideas than nominal brainstorming groups. METHOD EXPERIMENTAL DESIGN AND PROCEDURES The experimental design was an unbalanced repeated measures model with one between-group factor (Group History at two levels: ad hoc and established groups) and three within group factors (brainstorming technology at four levels: verbal, nominal, EBSanonymous, EBS-non-anonymous; Topic Sensitivity at two levels: high and low; and Contextual Cues at two levels: with and without Contextual Cues). 16 six-member established groups and 16 sixmember ad hoc groups participated in the study, each randomly assigned to 4 of the 16 (4 2 2) treatments. The full experimental design is depicted in Appendix A. As can be seen, this design ensured that each group brainstormed once with each brainstorming technology, twice with socially sensitive and twice with socially less sensitive topics, and twice under seeded and twice under unseeded conditions. Thus, each of the 32 groups was observed four times, for a total of 128 observations. The experimental sessions were identically structured and conducted by a facilitator who was unaware of the research hypotheses being investigated. The facilitator had excellent typing skills, had a good knowledge of the GroupSystemsV brainstorming software that was being used, and was provided with an experimental package containing precise instructions regarding how to conduct each session. When each group arrived at the experimental site, the facilitator presented herself and asked group members to introduce themselves. Group members were provided with name tags that were placed in front of each member and were visible to everyone. They were asked to sign a consent form and a confidentiality agreement and to fill out the presession questionnaire. This questionnaire inquired about the participants program of study, education

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 175 level, age, previous experience with brainstorming, work experience, previous experience with group work, typing speed, and computer experience. The presession questionnaires given to the established groups contained additional questions concerning how many months the group had already worked together, members position and tenure in the group, how many friends they had in the group, and the length of their friendships. After the group members filled out the prequestionnaire, the facilitator made a small presentation to describe how the session would unfold. In each of the four experimental tasks performed by each group, the following procedure was used. The facilitator provided the participants with verbal and written brainstorming instructions and rules (Bouchard & Hare, 1970; Osborn, 1957): The more ideas the better, the wilder the idea the better, do not criticize, and be as clear and concise as possible. Subsequently, a 3-minute brainstorming practice session was held on one of the four practice topics, using the brainstorming technology that the participants were going to use in the shortly upcoming experimental task. The practice topics were randomly assigned to the four experimental topics. At the end of the 3-minute warm-up period, the facilitator made sure that all participants were comfortable with using the brainstorming technology. Then, the group brainstormed for 15 minutes on one of the four experimental topics. At the end of the 15-minute session, participants were asked to respond to the postsession questionnaire. All 32 groups followed this procedure four times, once for each of the four experimental conditions they were subjected to. At the end of the fourth session, the facilitator conducted an open discussion on the experiment, thanked the participants for their participation, and paid each $20. These procedures and the presession and postsession questionnaires were initially tested with one ad hoc group and then were retested with four ad hoc groups. INDEPENDENT VARIABLES The four brainstorming technologies used in this study were operationalized as follows. In verbal brainstorming, group mem-

176 SMALL GROUP RESEARCH / April 2001 bers orally stated the ideas they generated, and these were keyboarded by a facilitator as the discussion occurred. To eliminate the potential confounding effect of typing versus other methods of idea recording, all ideas generated were keyboarded by the facilitator in all experimental conditions. 6 In nominal brainstorming sessions, each participant generated ideas individually and keyboarded them. They did not interact with other members of their group and did not have access to their ideas. In both EBS sessions, participants keyboarded the ideas they generated and could see on their screens a randomly selected subset of the ideas generated by the other members of the group. Although the subset of ideas randomly displayed on individual screens had no identifiers associated with them in the EBS-anonymous condition, in the EBS-non-anonymous condition, the software displayed the name of the participant who generated the idea. The operationalization of Topic Sensitivity was based on Gallupe, Cooper, Pollard, Cadsby (1996). Four topics two of which were thought to be socially sensitive (i.e., How can the spread of AIDS be reduced? and How can violent crimes be reduced?) and two of which were thought to be socially less sensitive (i.e., How can tourism be improved in Montreal? and How can the university parking be improved?) were used. As can be seen in Appendix A, all groups generated ideas on these topics in a counterbalanced order, so that each topic was used 8 times with each of the four technologies, 16 times without Contextual Cues, and 16 times in the presence of Contextual Cues. In addition, four practice topics were used: What benefits and difficulties would occur if everyone grew an extra thumb on each hand? What are some uses for a brick? What are some uses for a knife? What are the impacts of a longer university year? The presence or absence of supplemental environmental stimuli of a vague nature was used to operationalize Contextual Cues. In the condition where Contextual Cues were present (labeled idea seeding), a public screen in the room was used to display, for the duration of a particular brainstorming session, four ideas on the topic of that session. Participants were told that they were being

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 177 provided with these seed ideas as an inspirational aid. The displayed ideas, shown in Appendix B, were selected from among those previously generated by the research team and thought to provide a broad coverage on the brainstorming topic of the session along different dimensions (e.g., dimensions such as punishment, positive feedback, or control for the topic, How to Reduce Violent Crimes). It was thought that the presentation of ideas on different dimensions would provide additional environmental stimuli of a vague nature, similar to the superimposed images of Sobel and Rosenberg (1980). In the experimental sessions, Nonseeded groups were not provided with any such stimuli. The between-group factor, Group History, was operationalized through the use of 16 ad hoc and 16 established groups. The ad hoc groups were formed by randomly assigning 96 second- and thirdyear undergraduate business students to 16 groups. The established groups were 16 executive committees of different undergraduate student associations (also second- and third-year undergraduate students) from three faculties (management, engineering, industrial relations) of a large northeastern university. 7 Data collected from presession and postsession questionnaires showed that at the time of the experiment, established group members had worked together an average of 9 months (0 months in the case of ad hoc groups, difference p <.000) and that they had on average 3 friends in their group (less than 1 for ad hoc groups, difference p <.000). In addition, although established group members had been friends for more than 19 months, for ad hoc group members, some of whom happened to know one another by chance, this average was only 8 months (p <.000). Moreover, members of established groups had more group work experience than ad hoc groups (33 versus 19 times, p <.000). These results suggest that Group History was adequately operationalized. Finally, no significant typing speed, computer usage experience, and level of scholarship differences were found between established and ad hoc groups. 8 All 192 participants were familiar with microcomputers but had no prior EBS experience. The ad hoc groups were made up of 67 men and 29 women, whereas the established groups contained 51 men and 45 women.

178 SMALL GROUP RESEARCH / April 2001 DEPENDENT VARIABLES Three dependent variables were examined: total quality, mean quality, and number of good ideas. Scores for the three dependent variables were obtained with the following procedure. First, redundant ideas were eliminated within each of the 128 sets of ideas generated in the 128 experimental sessions to identify the set of unique ideas for each session. To do so, the procedure proposed by Bouchard and Hare (1970) was followed: Two treatment and hypothesis blind coders independently identified, for each experimental session, the unique ideas. The two coders then met to discuss and resolve their coding differences. A total of 128 sets of unique ideas were thus obtained (intercoder correlation for number of unique ideas per set r =.91, n = 128, p <.001). Second, to assess the quality of the unique ideas generated by each group, four questionnaires were developed (one per idea generation topic) using the following procedure. First, for each idea generation topic, each coder independently identified an overall list of unique ideas generated by all 32 groups. Then, the two coders met to compare their respective sets of unique ideas and resolve any differences. The intercoder agreement for this analysis was quite high with more than 90% of the unique ideas being common to the two lists. Next, one questionnaire per topic was developed, each containing all the unique ideas generated by the 32 groups on that particular topic. The number of unique ideas contained in each questionnaire was as follows: 249 for the AIDS topic, 211 for the Crime topic, 302 for the Tourism topic, and 366 for the Parking topic. For each unique idea in the questionnaire, three facets of quality were assessed on 7-point Likert-type scales, anchored as 1 (very low) and 7 (very high): the idea s originality, feasibility, and effectiveness. Diehl and Stroebe (1987) proposed that an idea s quality is reflected by its originality (the extent to which the idea is novel, out of the ordinary) and feasibility (the extent to which the idea is precise and the ease with which it can be implemented, given the current context such as the financial resources available, infrastructure, time required, legal issues, etc.). Although this conceptualiza-

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 179 tion of quality is satisfactory, numerous authors have argued that effectiveness (the extent to which the idea helps to solve the problem) is also an important dimension of quality (Ilgen & Klein, 1988; Muckler, 1982; Pritchard, 1990; Pritchard & Watson, 1992). It is argued here that although an idea may be very original and feasible, its overall quality cannot be very high if it does not help solve the problem at hand. Thus, the facet of effectiveness needs to be considered when evaluating idea quality. In the present study, this broader conceptualization of quality was used, and three facets of each unique idea (i.e., originality, feasibility, and effectiveness) was assessed. The experts. For each of the four topics, three independent professional experts, selected for their knowledge of the topic and their complementary expertise, were consulted. Each expert was first contacted by one of the researchers to describe the research project and to solicit their participation. The questionnaires were then delivered to each expert by a research assistant and picked up 2 weeks later. Because the questionnaires were quite long, in an effort to maintain the experts interest in rating the ideas, they were told to stop as frequently as they wished when responding to the questionnaire. They were also told that they could feel free not to respond to the questionnaire if they felt uncomfortable with the ideas expressed or if they felt that the task was too tedious. All 12 experts returned their questionnaires within 2 weeks, and they all indicated that the questionnaires, although long to fill, were interesting because they dealt with a subject that interested them. No significant response pattern was observed in the questionnaires. For each topic, the three respondents ratings were defined as being in agreement when they were within two points of each other. Table 3 presents the background of each expert and the percentage of ideas for which the respective pair or tri As can be seen in Table 3, the agreement scores between the experts were fairly high, especially when taking into account that the respondents were chosen for their complementary expertise on a particular topic and because they offered differing quality per-

180 SMALL GROUP RESEARCH / April 2001 spectives on that topic. For example, an epidemiologist, an infected patient, and a specialized physician rated the ideas generated for the AIDS topic. To the extent that the different experts have different priorities and use differing criteria in assessing idea quality, an agreement score between them is likely to be lower than if, for example, three epidemiologists would have assessed the quality of the ideas. For AIDS, the agreement scores ranged from 87% to 96% for the experts taken two at a time and was 78% for the three experts together (i.e., for 78% of the 249 unique ideas in the questionnaire, the three experts assessments were within 2 scale points of each other). For Crime, the experts agreed 88% to 97% of the time when analyzed as pairs and 80% for all three. For Parking, agreement scores ranged from 73% to 91% for pairs and was 62% for all three together. The agreement scores on the Tourism topic were lower than for the other topics, especially when all three respondents scores were analyzed together (33% agreement score). An examination of the agreement scores between pairs of experts indicates that the low agreement scores for Tourism are due to Expert 2 who was in low agreement both with Expert 1 (54%) and with Expert 3 (52%), whereas experts 1 and 3 were in high agreement (88%). This low agreement of Expert 2 with the other experts can probably be explained by that Experts 1 and 3 were both born and raised in the city and had been working there for a long time, whereas Expert 2 worked for a large international hotel chain and had only recently moved to the city. As a result, he could have been less familiar with local particularities, explaining his low agreement scores with the others. Although the agreement score for the three experts for the Tourism topic was low, we considered that it was preferable to keep the ratings of Expert 2 in the calculation of the overall quality scores rather than arbitrarily rejecting his or her assessments without sound theoretical reasons. It could be argued that Expert 2 provides a much needed international expertise that allows a better assessment of the quality of the ideas, especially in terms of their effectiveness. The quality scores. Consistent with past research (e.g., Diehl & Strobe, 1987; Gallupe et al., 1992), a quality score was calculated

Barki, Pinsonneault / BRAINSTORMING AND IDEA QUALITY 181 TABLE 3: Expert Agreement Scores per Topic Comparing Agreement Idea Generation Topics and Experts Experts Scores How to reduce the spread of AIDS 1-2.96 An epidemiologist specialized in AIDS infection 1-3.92 An AIDS-infected patient and leading researcher for the National Council for AIDS 2-3.87 A physician specialized in how AIDS is transmitted 1-2-3.78 How to reduce violent crimes 1-2.97 A criminologist, expert in violent crimes 1-3.96 A public attorney, specialized in violent crimes 2-3.88 A social psychologist specialized in violent crime prevention 1-2-3.80 How to improve university parking 1-2.91 The manager of parking for a major university 1-3.73 A principal urban planner of a city surrounding the university and affected by its parking regulations 2-3.83 The manager of the urban planning department of a major North American city in which the University is located 1-2-3.62 How to improve tourism in the city 1-2.54 The marketing vice-president of national hotel chain Number 1 1-3.88 The marketing and sales manager of a major international hotel chain 2-3.52 The marketing and sales vice-president of national hotel chain Number 2 1-2-3.33 NOTE: 1-2-3 represents the agreement between all three experts for that brainstorming topic, while 1-2 represents the agreement between Expert 1 and Expert 2 of that brainstorming topic, and so on, for example. for each unique idea by averaging its originality, feasibility, and effectiveness scores (assigned by the three experts of that topic). This resulted in an idea quality score for each unique idea listed in the questionnaire. Then, this score was assigned to each unique idea generated by a group on a particular topic. The procedure used to create quality scores at the group level is described below. Three different measures of idea quality at the group level were created: total quality, mean quality, and number of good ideas (Dennis et al., 1997; Diehl & Stroebe 1987, 1991; Stroebe & Diehl, 1994). Total quality was calculated by summing the quality scores of all unique ideas generated by a group. This score provides an overall picture of all the unique ideas generated by a group. However, it should be noted that a group s total quality score and the

182 SMALL GROUP RESEARCH / April 2001 number of unique ideas it generated are typically highly correlated (e.g., Diehl & Stroebe, 1987, 1991, found a correlation of.80 between the two measures) because the more unique ideas a group generates, the higher its total quality score. Mean quality was calculated by dividing the total quality score of a group with the number of unique ideas it generated and provides a quality score that is less correlated with the number of unique ideas generated by a group. Although mean quality is less dependent on the quantity of ideas, it does not say anything about the number of good ideas a group has generated. To capture this aspect, a third measure of quality, number of good ideas, was calculated. A good idea was defined as one having a quality score over a cutoff point, taken to be the overall average, for a given topic, of the quality scores of all the unique ideas generated by all groups. Number of good ideas therefore was scored for each group by counting the number of ideas generated by a group that had quality scores above the overall average (for that topic). Diehl and Stroebe (1987, 1991) and Dennis et al. (1997) provide arguments to the effect that number of good ideas for a group is the most appropriate quality measure because the main goal in brainstorming is to produce as many good ideas as possible (Rowatt et al., 1997). RESULTS The unbalanced repeated measures design with four independent variables (Technology, Group History, Topic Sensitivity, Contextual Cues) and three dependent variables (total quality, mean quality, number of good ideas) can be modeled as follows: Y ijklm = µ + α i + β j + χ k + δ l + (αβ) ij + (αχ) ik + (αδ) il + (βχ) jk + (βδ) jl + (χδ) kl + (αβχ) ijk + (αβδ) ijl + (αχδ) ikl + (βχδ) jkl + (αβχδ) ijkl + e ijklm. (1) Here, Y denotes the dependent variable, i denotes the level of the Technology factor, j denotes the level of the Group History factor, k denotes the level of the Topic Sensitivity factor, l denotes the level