Evaluation of Human Language Technologies
Report on the course
This thematic course was organized under the responsibility of the University of Copenhagen and in cooperation with ELDA (associate partner). It was held on 26-29 November 2012 on the premises of ELDA, 55-57 rue Brillat Savarin, 75013 Paris, France.
Goal: For any HLT research effort to be successful, it is essential that it be assessed through rigorous evaluations of the developed technologies. This allows performance benchmarking and a better understanding of possible limitations and challenging conditions. The workshop aimed at providing ESRs/ERs the background and skills to use and implement state-of-the–art evaluation tools and techniques for speech technologies, grammars and parsing, machine translation and speech-to-speech translation, information retrieval/filtering, multimodal interfaces, etc.
Focus: The workshop elaborated on the role of evaluation on the research progress, on the need for a truly European infrastructure for HLT evaluation, on the main reasons to promote an international dimension of the evaluation, insisting on the multilingual issues. The workshop introduced and described some evaluation concepts (comparative evaluation versus competition, technology evaluation versus usage/usability evaluation). It also described the different types of evaluation and how to ensure that evaluation does not kill innovative not-yet-mature approaches.
Audience: Given the interdisciplinary nature of the workshop, it was aimed at any ESRs/ERs involved in the development of algorithms and systems for speech technologies, machine translation, parsers, information retrieval, multimodal interfaces, etc. Twelve researchers participated in the course, of which eight were CLARA fellows and four were external.
Procedure: The workshop was given as a combination of presentations, group tutorials and hands-on training in the use of evaluation technologies for HLT.