Annotation Projection for Discourse Connectives

Transcription

1 SFB 833 / Univ. Tübingen Penn Discourse Treebank Workshop

2 Annotation projection Basic idea: Given a bitext E/F and annotation for F, how would the annotation look for E? Examples: Word Sense Disambiguation (Bentivogli and Pianta, 2005) FrameNet frames (Pado and Lapata, 2005) Coreference chains (Postolache et al., 2006) Ways of realization: Experiments in vitro Adding manual annotation to a parallel corpus Translating an annotated corpus Automatic annotation of a parallel corpus

3 Annotation Experiment Training on Penn Discourse Treebank: Identification, desambiguation of discourse connectives (Pitler and Nenkova, 2009) Finding connective arguments (Wellner, 2009) Automatic annotation EuroParl-EN Training on projection from EuroParl-DE Testing on manual annotated material from TüBa-D/Z

4 Automatic Annotation: Example

5 Automatic Annotation Potential Connectives e.g. and, either... or, as soon as Discourse Connective yes/no Classification into discourse relation using syntactic features Connective string node, parent, left/right neighbour VP in right neighbour? traces, function labels (Pitler and Nenkova, 2009) here: tense/mood of connected sentences

6 Automatic annotation (2) Anaphoricity Finding Connective Arguments Path between connective and arguments Constituents (IN SBAR S) Dependencies+labels ( ccomp advcl) fork production SBAR 1 IN C S 2

7 Automatic Annotation (3) In-Domain (PTB) Identification: 92% P, 98% R Top-Level Relations: 94-95% Medium Taxonomy Level: 84% Arguments: Anaphoricity F=0.975 Arg1 87% Head, 76% PS node Arg2 93% Head, 89% PS node

8 Automatic Annotation (3) In-Domain (PTB) Identification: 92% P, 98% R Top-Level Relations: 94-95% Medium Taxonomy Level: 84% Arguments: Anaphoricity F=0.975 Arg1 87% Head, 76% PS node Arg2 93% Head, 89% PS node Out of Domain (EuroParl) Automatic Parses (Berkeley Parser) Identification: 83% P, 97% R

9 Error analysis OK: I regret this, but the vote has already been taken and [ arg1 the decision is made] [ conn so] [ arg2 let us leave the matter there]. Ich bedauere das, aber die Abstimmung ist durchgeführt worden, [ arg1 die Entscheidung ist gefallen], [ conn also] [ arg2 lassen wir die Dinge]. [ Contingency.Cause.Result ] Misparse in EN: [ arg1 Madam President, [ conn if] [ arg2 the vote records correctly how my Group voted I shall not, and cannot], object to that]. Frau Präsidentin, [ conn wenn] [ arg1/arg2aus der Abstimmung einwandfrei hervorgeht, wie meine Fraktion abgestimmt hat, dann werde und kann ich nichts dagegen sagen]. [ Contingency.Condition.Hypothetical ]

10 Projection Properties of Discourse Connectives Syntactically heterogeneous (Subordinating/coordinating) conjunctions and, after Adverbial PPs (for example) Discontinuous units (if... then, either... or; as soon as) No content words unreliable word alignment

11 CRF-Baseline We will be in a position to judge just as soon as the Commission pres Über Prüfsteine werden wir alsbald verfügen, wenn die Kommission ihr R B-DC B-DC

12 CRF Baseline Each word that is aligned with a connective Direct sequence classifier Word, POS, starting/ending clauses With intersect alignment: P=0.839 R=0.382 F=0.525 With grow-diag-final alignment: P=0.740 R=0.419 F=0.535

13 Lexicon-based List of connectives from the Handbuch der deutschen Konnektoren (Pasch et al., 2003) Baseline: treat all entries as discourse connectives P=0.270 R=0.940 F=0.421

14 Alignment+lexicon We will be in a position to judge just as soon as the Commission pres Über Prüfsteine werden wir alsbald verfügen, wenn die Kommission ihr R

15 Lexicon-based (2) Positive examples: aligned with a discourse connective on English side Negative examples: Unaligned or aligned with a non-discourse occurrence Syntactic features (German): Label of node, parent, left/right neighbour node Discriminative parser (Versley and Rehbein, 2009) P=0.623 R=0.765 F1=0.687

16 Lexicon-based (3) Why is the lexicon-based approach better? Cleaner training data? Better features? Additional experiments: CRF with cleaner training data Replace HdK lexicon with induced lexicon

17 Lexicon induction Candidates: Any sequence occurring as counterparts of English DCs Average Overlap metric: Dice coefficient between candidate string and mapped DC, averaged over all aligned occurrences

18 HdK vs. induced abermals abgesehen davon allenfalls abschließend allerdings als als Beispiel als Ergebnis als Folge also als ob am Ende anders ausgedrückt

19 Results overview Prec Recl F β=1 all HdK simple CRF, giza-refined simple CRF, giza-intersect HdK+classifier induced+classifier HdK+CRF induced+crf

20 Summary Projected training data + lexicon Results in usable classifier for discourse connectives Need evaluation data for relation labels, argument finding Combine with hand-annotated data in target language

21 Thanks for listening! THE END

22 Alignment example: f e, e f Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly

23 Alignment example: Union/Intersection Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly

24 Alignment example: Heuristic refinement Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly

25 Bentivogli, L. and Pianta, E. (2005). Exploiting parallel texts in the creation of multilingual semantically annotated resources: the MultiSemCor corpus. Natural Language Engineering, 11(3): Carlson, L., Marcu, D., and Okurowski, M. E. (2003). Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current Directions in Discourse and Dialogue. Kluwer. Pado, S. and Lapata, M. (2005). Cross-lingual projection of role-semantic information. In Proceedings of HLT/EMNLP Pasch, R., Brauße, U., Breindl, E., and Waßner, U. H. (2003). Handbuch der deutschen Konnektoren. Walter de Gruyter, Berlin / New York. Pitler, E. and Nenkova, A. (2009). Using syntax to disambiguate explicit discourse connectives in text. In ACL 2009 short papers. Postolache, O., Cristea, D., and Orasan, C. (2006). Transferring coreference chains through word alignment. In Proceedings of

26 the 5th Language Resources and Evaluation Conference (LREC2006). Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The Penn Discourse Treebank 2.0. In Proceedings of LREC Versley, Y. and Rehbein, I. (2009). Scalable discriminative parsing for German. In Proc. IWPT Wellner, B. (2009). Sequence Models and Ranking Methods for Discourse Parsing. PhD thesis, Brandeis University. Wolf, F. and Gibson, E. (2005). Representing discourse coherence: A corpus-based study. Computational Linguistics, 31(2):