SFB 833 / Univ. Tübingen Penn Discourse Treebank Workshop
Annotation projection Basic idea: Given a bitext E/F and annotation for F, how would the annotation look for E? Examples: Word Sense Disambiguation (Bentivogli and Pianta, 2005) FrameNet frames (Pado and Lapata, 2005) Coreference chains (Postolache et al., 2006) Ways of realization: Experiments in vitro Adding manual annotation to a parallel corpus Translating an annotated corpus Automatic annotation of a parallel corpus
Annotation Experiment Training on Penn Discourse Treebank: Identification, desambiguation of discourse connectives (Pitler and Nenkova, 2009) Finding connective arguments (Wellner, 2009) Automatic annotation EuroParl-EN Training on projection from EuroParl-DE Testing on manual annotated material from TüBa-D/Z
Automatic Annotation: Example
Automatic Annotation Potential Connectives e.g. and, either... or, as soon as Discourse Connective yes/no Classification into discourse relation using syntactic features Connective string node, parent, left/right neighbour VP in right neighbour? traces, function labels (Pitler and Nenkova, 2009) here: tense/mood of connected sentences
Automatic annotation (2) Anaphoricity Finding Connective Arguments Path between connective and arguments Constituents (IN SBAR S) Dependencies+labels ( ccomp advcl) fork production SBAR 1 IN C S 2
Automatic Annotation (3) In-Domain (PTB) Identification: 92% P, 98% R Top-Level Relations: 94-95% Medium Taxonomy Level: 84% Arguments: Anaphoricity F=0.975 Arg1 87% Head, 76% PS node Arg2 93% Head, 89% PS node
Automatic Annotation (3) In-Domain (PTB) Identification: 92% P, 98% R Top-Level Relations: 94-95% Medium Taxonomy Level: 84% Arguments: Anaphoricity F=0.975 Arg1 87% Head, 76% PS node Arg2 93% Head, 89% PS node Out of Domain (EuroParl) Automatic Parses (Berkeley Parser) Identification: 83% P, 97% R
Error analysis OK: I regret this, but the vote has already been taken and [ arg1 the decision is made] [ conn so] [ arg2 let us leave the matter there]. Ich bedauere das, aber die Abstimmung ist durchgeführt worden, [ arg1 die Entscheidung ist gefallen], [ conn also] [ arg2 lassen wir die Dinge]. [ Contingency.Cause.Result ] Misparse in EN: [ arg1 Madam President, [ conn if] [ arg2 the vote records correctly how my Group voted I shall not, and cannot], object to that]. Frau Präsidentin, [ conn wenn] [ arg1/arg2aus der Abstimmung einwandfrei hervorgeht, wie meine Fraktion abgestimmt hat, dann werde und kann ich nichts dagegen sagen]. [ Contingency.Condition.Hypothetical ]
Projection Properties of Discourse Connectives Syntactically heterogeneous (Subordinating/coordinating) conjunctions and, after Adverbial PPs (for example) Discontinuous units (if... then, either... or; as soon as) No content words unreliable word alignment
CRF-Baseline We will be in a position to judge just as soon as the Commission pres Über Prüfsteine werden wir alsbald verfügen, wenn die Kommission ihr R B-DC B-DC
CRF Baseline Each word that is aligned with a connective Direct sequence classifier Word, POS, starting/ending clauses With intersect alignment: P=0.839 R=0.382 F=0.525 With grow-diag-final alignment: P=0.740 R=0.419 F=0.535
Lexicon-based List of connectives from the Handbuch der deutschen Konnektoren (Pasch et al., 2003) Baseline: treat all entries as discourse connectives P=0.270 R=0.940 F=0.421
Alignment+lexicon We will be in a position to judge just as soon as the Commission pres Über Prüfsteine werden wir alsbald verfügen, wenn die Kommission ihr R
Lexicon-based (2) Positive examples: aligned with a discourse connective on English side Negative examples: Unaligned or aligned with a non-discourse occurrence Syntactic features (German): Label of node, parent, left/right neighbour node Discriminative parser (Versley and Rehbein, 2009) P=0.623 R=0.765 F1=0.687
Lexicon-based (3) Why is the lexicon-based approach better? Cleaner training data? Better features? Additional experiments: CRF with cleaner training data Replace HdK lexicon with induced lexicon
Lexicon induction Candidates: Any sequence occurring as counterparts of English DCs Average Overlap metric: Dice coefficient between candidate string and mapped DC, averaged over all aligned occurrences
HdK vs. induced abermals abgesehen davon allenfalls abschließend allerdings als als Beispiel als Ergebnis als Folge also als ob am Ende anders ausgedrückt
Results overview Prec Recl F β=1 all HdK 27.0 94.9 42.1 simple CRF, giza-refined 74.0 41.9 53.5 simple CRF, giza-intersect 83.9 38.2 52.5 HdK+classifier 62.3 76.5 68.7 induced+classifier 58.3 56.6 57.5 HdK+CRF 74.7 43.4 54.9 induced+crf 70.2 43.4 53.6
Summary Projected training data + lexicon Results in usable classifier for discourse connectives Need evaluation data for relation labels, argument finding Combine with hand-annotated data in target language
Thanks for listening! THE END
Alignment example: f e, e f Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly
Alignment example: Union/Intersection Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly
Alignment example: Heuristic refinement Vielen Dank, Herr Segni, das will ich gerne tun. Thank you, Mr Segni, I shall. do sogladly
Bentivogli, L. and Pianta, E. (2005). Exploiting parallel texts in the creation of multilingual semantically annotated resources: the MultiSemCor corpus. Natural Language Engineering, 11(3):247 261. Carlson, L., Marcu, D., and Okurowski, M. E. (2003). Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current Directions in Discourse and Dialogue. Kluwer. Pado, S. and Lapata, M. (2005). Cross-lingual projection of role-semantic information. In Proceedings of HLT/EMNLP 2005. Pasch, R., Brauße, U., Breindl, E., and Waßner, U. H. (2003). Handbuch der deutschen Konnektoren. Walter de Gruyter, Berlin / New York. Pitler, E. and Nenkova, A. (2009). Using syntax to disambiguate explicit discourse connectives in text. In ACL 2009 short papers. Postolache, O., Cristea, D., and Orasan, C. (2006). Transferring coreference chains through word alignment. In Proceedings of
the 5th Language Resources and Evaluation Conference (LREC2006). Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The Penn Discourse Treebank 2.0. In Proceedings of LREC 2008. Versley, Y. and Rehbein, I. (2009). Scalable discriminative parsing for German. In Proc. IWPT 2009. Wellner, B. (2009). Sequence Models and Ranking Methods for Discourse Parsing. PhD thesis, Brandeis University. Wolf, F. and Gibson, E. (2005). Representing discourse coherence: A corpus-based study. Computational Linguistics, 31(2):249 287.