1 Introduction We have been carrying out an evaluation of the eects of teaching logic with Barwise and Etchemendy's Hyperproof [1]. One of our major n

Transcription

1 Hyperproof: the multimodal moral Jon Oberlander Keith Stenning Richard Cox April 18, 1996 Co-ordinates: Correspondent Jon Oberlander J.Oberlandered.ac.uk Pmail Human Communication Research Centre, University of Edinburgh, 2 Buccleuch Place, Edinburgh, EH8 9LW, Scotland Phone Fax

2 1 Introduction We have been carrying out an evaluation of the eects of teaching logic with Barwise and Etchemendy's Hyperproof [1]. One of our major ndings has been that individual dierences between students have a signicant eect on students' responses to Hyperproof: their prior cognitive style inuences both the overall eectiveness of the teaching regime, and the actual proof structures that students produce under exam conditions [9, 7, 8]. Now, Barwise and Etchemendy designed Hyperproof to support heterogeneous reasoning, in which information from diering modalities sentential and graphical is combined, or transferred from one modality to another. It is obviously, therefore, a multimodal system containing a visual sub-system. But given that one group of students benets particularly from being taught with Hyperproof, we can ask: do they do well because it is a visual logical system, or do they do well because it is multimodal? We here list some of Hyperproof's special rules, before sketching the way we split our student population into two cognitive style groups; we also indicate the methods we have recently been pursuing in our investigations: `proofograms', and corpus-based bigram analysis. We can then discuss whether the evidence favours the view that those who benet most from Hyperproof do so on account of its visual nature, or because of its multimodality. 2 Cognitive styles The main study compared two groups of subjects; one group ( n = 22 at course end) attended a one-quarter duration course taught using the heterogeneous reasoning approach of Hyperproof. A comparison group ( n = 13 at course end) were also taught for one quarter, but in the traditional syntactic manner supplemented with exercises using a graphics-disabled version of Hyperproof (to control for the motivational and other eects of computer-based activities). A fuller description of the method and procedure is provided elsewhere [9]. Subjects were administered two kinds of pre- and post-course paper and pencil test of reasoning. One test was of `analytical reasoning' and contained two kinds of item derived from the GRE-type of scale of that name [5, 7]. We refer to this test as the ` reasoning/argument analysis. GRE ' test. The rst subscale consists of verbal The other subscale consists of items often 2

3 Apply Extracts information from a set of sentential premises; expresses it graphically Assume Observe Inspect Introduces a new assumption into a proof, either graphically or sententially Extracts information from the situation; expresses it sententially Extracts common information from a set of cases; expresses it sententially Merge Extracts common information from a set of cases; expresses it graphically Close Declares that a sentence is inconsistent with either another sentence, or the current graphical situation CTA (Check truth of assumptions) Declares that all sentential and graphical assumptions are true in the current situation Exhaust Declares that a part of a proof exhausts all the relevant cases Figure 1: A set of relevant Hyperproof rules. best solved by constructing an external representation of some kind (such as a table or a diagram). We label these subscales as `indeterminate' and `determinate', respectively. Scores on the determinate subscale of the GRE test were used to classify subjects within both Hyperproof and syntactic groups into DetHi and DetLo sub-groups. The score reects subjects' facility for solving a type of item that often is best solved using an external representation; DetHi scored well on analytical reasoning items; DetLo scored less well on such items. Loosely, we may consider DetHi subjects to be more `diagrammatic', and DetLo to be less so. DetHi and DetLo subjects in the Hyperproof and syntactic groups responded dierently to traditionally versus heterogeneously taught courses [2, 9]. In essence, DetHi students benetted from Hyperproof, and did less well with traditional teaching; DetLo students did better under the traditional regime. 3 Hyperproof rules Hyperproof supports the use of both the traditional syntactic rules of rstorder logic, and special graphical rules. The most important of these are summarised in Figure 3; see Barwise and Etchemendy [1] for a full account of Hyperproof's rule system. 3

4 Figure 2: Proofograms for two subjects attempting an indeterminate question.left: C2, a DetLo subject; right: C14, a DetHi subject. Proof steps are plotted on the x-axis; the concreteness of the current graphical situation is computed for each step of the proof, and is plotted on the y-axis. Horizontal lines indicate dependency structure; vertical lines indicate uses of Assume; sloping lines indicate uses of Apply or Merge. The visual dierence between the two proofograms is striking. C2's proof is `spikey', indicating a series of independent, concrete cases. C14's proof is `layered', indicating parallel sub-case structures with abstract superordinate cases. 4 Proofograms and bigrams Both groups sat post-course, computer-based Hyperproof exams. The four questions set the Hyperproof group contained two types of item: determinate and indeterminate. Here, determinate problems were taken to be those which did not utilise Hyperproof's abstraction conventions for objects' spatial or visual attributes. As well as concrete depictions of objects, Hyperproof allows `graphical abstraction symbols', which leave attributes under-specied: the cylinder depicts objects of unknown size; the paper bag depicts objects of unknown shape. Student-computer interactions were dynamically logged. The logs were time stamped and permitted a full, stepby-step, reconstruction of the time course of the subject's reasoning. 4.1 Proofograms We can score each step of each proof on the basis of number of concrete situations compatible with the graphical depiction; one possible scoring method is described elsewhere [7]. A low score indicates more abstraction; a higher score indicates more concreteness. As an exploratory method, we can then graph the way concreteness varies through the course of a proof, and relate this to the hierarchical structure of the proof. We call such graphs `proofograms'. Figure 2 shows how subjects C2 and C14 tackle an indeterminate exam question. The visual dierences between proofograms are quite striking: one group is `spikey', and the other is `layered'. The dierences are particularly pronounced on indeterminate questions, and Q-sort tests indicate that these questions reliably elicit layered proofs from DetHi subjects, and spikey proofs from DetLo [8]. The basic message appears to be that DetHi subjects' layered proofs involve 4

5 the gradual addition of concreteness to graphical situations, whereas DetLo subjects' spikey proofs involve an immediate transition to full concreteness. 4.2 Bigrams (and trigrams) If proofs are considered to be hierarchically structured discourses, then it is possible to analyse them using techniques developed for the study of natural language corpora [4]. In particular, we can carry out bigram and trigram analyses of rule use, where hierarchy and linear ordering can be taken into account [6, 8]. These analyses have taken into account the spikey/layering distinction: since Assume is the most frequent means of adding detail to cases, it is worth distinguishing between uses of the rule which introduce totally concrete graphical situations, and those which leave some abstractness in the graphic. We have used Fullassume to denote invocations of Assume that introduce situations of maximal concreteness, and assume to denote all other invocations. An important nding is that on determinate questions, there is a strong and highly signicant correlation between DetHi and DetLo bigram proles. By contrast, on indeterminate questions, the correlation is weak and nonsignicant. Hence, we can say with reasonable condence that indeterminate questions are better at discriminating the proles of the two subject groups. This is again consistent with the idea that the groups respond to these questions in rather dierent ways. 5 Visuality versus multimodality Indeterminate problems place almost all the initial, given information in the calculus window pane; to solve such problems, graphical representations must be constructed, and since so little is pre-specied in the graphical window pane, subjects have considerable freedom over the precise path they follow. Why do the subject groups diverge? One tempting hypothesis comes from identifying our DetHi DetLo distinction with the traditional visualiser verbaliser distinction. On this view, the diagrammatically capable DetHi subjects are just the visualisers, and therefore, they prefer to use the graphical modality when it is available. The diagrammatically less capable DetLo are the verbalisers, and hence prefer the sentential modality or at least, they do not show a strong preference for the graphical. 5

6 Table 1: Use of diering types of Assume RULE assume Fullassume DetHi DetLo frequency percentage average frequency percentage average of use per subject of use per subject An alternative explanation is that DetHi subjects are better at multimodal reasoning, mixing sentential and graphical information. On this account, DetLo might be perfectly happy in the graphical modality, so long as they do not have to translate information back and forth between the graphical and the sentential. The data provide supercial support for the initial `visual preference' hypothesis, but in each case we can show that the alternative `multimodal' hypothesis is a better explanation. First, consider the way that use of assume and Fullassume varies between the DetHi and DetLo groups, as shown in Table 1. DetHi make more use of assume than DetLo; the latter make more use of Fullassume than the former. The bigram assume Fullassume is found to be signicant in DetHi indeterminate proofs, but not in DetLo proofs. However, these facts do not favour the view that DetHi prefer graphics over language. On the contrary, DetLo subjects' favouring of Fullassume over assume indicates that they make a more rapid transition to the graphical modality. They seem to translate information from sentential into graphical in one step, and to continue working with maximally concrete graphical situations thereafter. In a sense, they are exhibiting a preference for the graphical modality. The two groups do seem to use Assume in rather dierent ways, particularly on questions containing graphical abstraction. DetHi use it to construct completely concrete situations in stages. As they do so, they are adding information to the graphical window, either by assumption, or by transferring it from the sentential window (via Apply). But the key point is that both groups are `preferring' graphics here; they dier over the way they use them. The DetHi group operate over the graphical situations, rather than just outputting them. In their case, a graphical situation is likely to act as input to 6

7 further stages of proof construction; for the DetLo group, however, graphics are not augmented by the addition of further information. Secondly, we might consider that support for the hypothesis of visual preference comes from the pattern associated with the bigram Inspect Merge. This is found to be signicant in DetLo subjects' indeterminate proofs, but nowhere else. The rules Inspect and Merge are used following a set of cases. The former expresses common information from these cases sententially. The latter expresses this common information graphically. DetHi and DetLo seem to dier over the order in which they invoke these rules. Table 2 indicates the dierent frequencies of the given trigrams across all proofs. It Table 2: Use of diering orders for Merge versus Inspect TRIGRAM Exhaust Inspect Merge Exhaust Merge Inspect frequency per group DetHi DetLo might be argued, then, that the DetLo group is as happy with graphical expressions as with sentential expressions. On this view, the DetHi group exhibits a very strong preference for rst abstracting from a set of cases, expressing this information graphically. Only then do they express the abstraction sententially. However, it is worth observing that use of the trigram Exhaust Inspect Merge demonstrates a misunderstanding of the use of Merge: after an Inspect it is actually redundant; people who deploy it in this way cannot actually use its results in subsequent proof construction. Hence, if DetLo are using it as often as they use Exhaust Merge Inspect, then half the time, they are proceeding directly from a set of concrete situations to the sentential expression, without essentially using the graphical Merge operation. This is further evidence for the multimodal hypothesis: while DetHi subjects manipulate their graphics in meaningful ways to mediate between modalities, DetLo subjects prefer to work in one modality rather than the other, without showing their translations. Finally, in comparing the two explanations, we can consider the fact that DetHi subjects make much more use of rules that transfer information between the modalities, as shown in Table 3. As part of their staged approach to graphical concreteness, DetHi make the bigram assume Apply 7

8 Table 3: Use of inter-modal rules RULE frequency per group frequency per subject DetHi DetLo DetHi DetLo Apply Observe an important part of their proofs. It appears signicantly distributed in both determinate and indeterminate questions. In addition, DetHi use the bigram much more frequently than DetLo: 44 times to just 7. By contrast, DetLo subjects exhibit a greater tendency to invoke Apply as the rst rule in their proofs (thereby giving rise to the bigram Given Apply). Subsequent interaction between the modalities is thereby reduced, with all work being performed within the graphical window, constructing cases. 6 Conclusion In summary, we nd most support for the multimodal hypothesis: the DetHi group seem to be more able to translate information between the modalities, using abstract graphical representations of the logical situations to construct incrementally their proof. DetHi subjects assume a more dynamic relationship between the modalities DetLo choose to move all relevant information to the graphical window initially, whereas DetHi subjects perform this transfer in phases. So, we already know that DetHi benet most from being taught with Hyperproof. It now seems that the reason their proofs dier from those of their DetLo counterparts lies in the multimodal, rather than visual, nature of their reasoning. We can therefore conclude that Hyperproof's virtue when it is virtuous is not its visuality. Rather, Hyperproof's moral is its multimodality. 7 Acknowledgements The support of the Economic and Social Research Council for HCRC gratefully acknowledged. The work was supported by UK Joint Councils is 8

9 Initiative in Cognitive Science and HCI, through grant G (Signal: Specicity of Information in Graphics and Natural Language); and by NATO Collaborative research grant (Cognitive Evaluation of Hyperproof). The rst author is supported by an EPSRC Advanced Fellowship. Special thanks to John Etchemendy, Tom Burke and Mark Greaves at Stanford, and Chris Brew at HCRC. This version of the paper includes material to be reported in Oberlander et al. citeober-cl. The conference presentation would focus on new results on relative orderliness in proof styles, as yet unreported. References References [1] Barwise, J. and Etchemendy, J Hyperproof. csli Lecture Notes. Chicago: Chicago University Press. [2] Cox, R., Stenning, K. and Oberlander, J Graphical eects in learning logic: reasoning, representation and individual dierences. In Proceedings of the 16th Annual Meeting of the Cognitive Science Society, pp237{242, Atlanta, Georgia, August. [3] Cox, R., Stenning, K. and Oberlander, J The eect of graphical and sentential logic teaching on spontaneous external representation. Cognitive Studies: bulletin of the Japanese Cognitive Science Society, 2(4), 1{20. [4] Dunning, T Accurate methods for the statistics of surprise and coincidence. Computational Linguistics 19, 61{74. [5] Duran, R., Powers, D. and Swinton, S Construct Validity of the GRE Analytical Test: A Resource Document. ets Research Report 87{11. Princeton, NJ: Educational Testing Service. [6] Monaghan, P A corpus-based analysis of individual dierences in proof-style. MSc Thesis, Centre for Cognitive Science, University of Edinburgh. [7] Oberlander, J., Cox, R. and Stenning, K. in press. Proof styles in multimodal reasoning. To appear in Seligman, J. and Westerstahl, D. 9

10 (Eds.) Language, Logic and Computation: The 1994 Moraga Proceedings. Stanford: CSLI Publications. [8] Oberlander, J., Monaghan, P., Cox, R., Stenning, K. and Tobin, R. submitted Unnatural language discourse: an empirical study of multimodal proof styles. Submitted to Computational Linguistics. [9] Stenning, K., Cox, R. and Oberlander, J Contrasting the cognitive eects of graphical and sentential logic teaching: reasoning, representation and individual dierences. Language and Cognitive Processes, 10, 333{