Abstracts | Program

Invited talk: Lluís Màrquez (Departament de Llenguatges i Sistemes Informàtics, Universitat Politècnica de Catalunya, Barcelona, Spain)

Approaches and Evaluation of Automatic Systems for Semantic Role Labeling: Experiences at CoNLL and SemEval

Semantic role labeling (SRL), the computational identification and labeling of arguments in text, has become a leading task in computational linguistics today. The recent availability of large resources and the development of statistical machine learning methods have heightened the amount of effort in this field. In parallel, several initiatives have been carried out to evaluate and compare SRL systems under fixed experimental settings. We find up to seven different evaluation exercises in the CoNLL shared tasks of 2004 and 2005, Senseval-3, and SemEval-2007. In this talk I will start by introducing the SRL task, the major resources that are being used, and the general architecture of computational systems. Then, I will move to the analysis and description of the key issues and results in semantic role labeling. This part will be guided by the experiences and lessons learnt in the abovementioned international evaluations. I will put special emphasis on assessing weaknesses in semantic role labeling and identifying important challenges for the near future in the field.


Gerwert Stevens (GridLine B.V., Amsterdam, The Netherlands) and Paola Monachesi (Utrecht Institute of Linguistics, Universiteit Utrecht, Utrecht, The Netherlands)

Pioneering approach to Dutch semantic role labeling

We present an approach to automatic semantic role labeling (SRL) carried out in the context of the Dutch Language Corpus Initiative (D-Coi) project. Adapting earlier research - which has mainly focused on English - to the Dutch situation poses an interesting challenge, especially because there is no semantically annotated Dutch corpus available that can be used as training data. Our automatic SRL approach consists of three steps:
  • bootstrapping from a syntactically annotated corpus by means of a rule based tagger developed for this purpose
  • manual correction on the basis of the PropBank guidelines which have been adapted to Dutch
  • Training a memory-based classifier on the manually corrected data.
The result is a Dutch corpus annotated with semantic roles, and a set of annotation guidelines. Both of which provide a solid base for future research.


Roser Morante (ILK Research Group, Tilburg University, Tilburg, The Netherlands)

Dependency-based semantic role labeling

In this talk I present several semantic role labelers of Spanish developed for the project "Semi-automatic techniques for semantic role labeling applied to a Spanish corpus" financed by the Spanish Ministry of Education. Semantic role labeling is a sentence-level natural-language processing task in which semantic roles are assigned to all arguments of a predicate. The main components of the systems that I present are memory-based classifiers. Memory-based language processing (Daelemans and Van den Bosch 2005) is based on the idea that NLP problems can be solved by storing annotated examples of the problem in their literal form in memory, and applying similarity-based reasoning on these examples in order two solve new ones.

The systems differ in the type of information that they are based on: constituent syntax information versus dependency syntax, gold standard syntax information versus predicted syntax, syntactic information versus no syntactic information. I will pay special attention to the systems that use dependency syntax information in some way, I will compare their results, and discuss their limitations.

W. Daelemans and A. Van den Bosch (2005) Memory-based language processing. Cambridge, UK: Cambridge University Press.


Sander Canisius (ILK Research Group, Tilburg University, Tilburg, The Netherlands)

Memory-based dependency parsing with constraint satisfaction inference

Triggered by the growing interest in machine learning for structured output spaces, dependency parsing has recently received wide attention as a target for machine learning. From a machine learning perspective, the task is difficult because of the many global interactions that exist between dependency relations within the same sentence. These interactions can only successfully be dealt with by classifiers that actively try to optimise the quality of the complete output structure, rather than only that of individual dependency relations.

In this talk I will present a generic framework for structured prediction based on constraint satisfaction, and show how it is applied to dependency parsing. The technique is based on standard classifiers that have been trained to predict constraints on the output space. A constraint satisfaction solver is then used to find the output structure that best satisfies those constraints. Using dependency parsing for illustration, I will give examples of the type of constraints that can be used, and how they contribute to the quality of the end result.

Khalil Sima'an (Institute for Logic, Language & Computation, University of Amsterdam, Amsterdam, The Netherlands)

Statistical parsing: New challenges and application perspectives

The best parsing systems produce f-score results of around 90% on newspaper text from the Penn WSJ treebank. Impressive as this may seem, especially when contrasted with the results of linguistic work from the 1970's and 1980's, the state-of-the-art parsing models will not adapt to new situations as easy as it may seem. One obvious problem is that the models are specific for English and will not parse as well languages with freer-word order and/or richer morphology than English, e.g. for Semitic languages such as Hebrew and Arabic. Another problem is the fact that a parser trained on the Penn treebank will not work as well for other domains, let alone that such domain-specific parsers (trained on specifically built treebanks) can be combined into a single system to parse as accurately within open domain situations such as the web. Regardless of these parsing issues, there is a more urgent question as to whether parsing may be of any use in actual applications, such as Machine Translation and Language Modeling. In this talk I will outline the main idea of my VIDI project, that parsing should be viewed as it was originally intended, as a mechanism for describing statistical regularities in the data. In this view, the syntactic structure of an utterance is an unobserved distribution over a set of structures. Consequently, parsing should not aim at selecting a single linguistic structure, instead it should aim at providing a better vehicle for describing statistical, structural properties of samples of language utterances. Starting out from this hypothesis I will summarize ongoing work on parsing Semitic languages, on incorporating subdomain sensitivity into existing parsers as a vehicle to study how to combine subdomain parsers into a single system, and on employing syntactic structure in improving large scale statistical machine translation systems.


Menno van Zaanen (ILK Research Group, Tilburg University, Tilburg, The Netherlands)

Unsupervised Learning: Syntax or Semantics?

A grammar is a finite structure that describes a language, a possibly infinite set of sentences. Grammatical inference deals with automatically building a grammar from a sample of sentences taken from a language. When applying grammatical inference to natural language, the intuitive aim is to learn the syntax of the language. However, it turns out that the learned grammars have semantic properties as well as the expected syntactic properties. Evaluation of grammatical inference output is typically performed on syntactic structures, which means that the semantic aspects may have a negative impact on the measured performance of the system. To resolve this problem, I propose to discard the evaluation on syntactic structures and instead evaluate in context, which measures the actual use or benefit of the learned structure.


Walter Daelemans (Centre for Dutch Language and Speech, Universiteit Antwerpen, Antwerp, Belgium)

Domain Adaptation in Memory-Based Shallow Parsing

Domain adaptation is probably the single most important current problem for Natural Language Processing systems based on supervised Machine Learning. Whenever the distributions of training and test data differ considerably, performance of supervised learning based systems degrades in an appalling way. In this talk we provide an overview of the problem and describe the 'simple but effective" way of solving it: retraining of modules using annotated data from the new domain. We explain how this works for the adaptation of an existing Wall Street Journal trained shallow parser to biomedical language. However, this retraining approach is fundamentally unsatisfactory. We review a number of suggestions for semi-automatic domain adaptation, none of which seems to work better than simply hand-annotating a small sample of the new domain. We end with some more optimistic news from computational psycholinguistics in morphology, showing that in a supervised learning method like memory-based learning, not optimizing crucial parameters like the value of k on the training data (implementing a lexicon reconstruction task), allows better transfer to tasks where the goal is matching participant behaviour. In other words, better transfer is achieved by not optimising k. We present some preliminary work on a similar approach to domain adaptation.




Last update: Thu Jan 16 2008