VERTa: a machine translation evaluation metric

Applications to L2 research on Spanish and English

Abstract

Abstract: This article presents VERTa (https://github.com/jatserias/VERTa for the full version and http://grial.ub.edu:8080/VERTaDemo/ for the Spanish online demo), a machine translation (MT) evaluation metric for English and Spanish. VERTa uses linguistic information to evaluate machine-translated sentences by comparing them with sentences translated by human translators. Unlike other metrics, VERTa provides not only a score for each sentence compared, but also a more qualitative analysis of the results obtained. This article discusses the steps carried out before designing and implementing the metric: the linguistic study of the development corpus to find the most relevant linguistic features that the metric should be able to cover, and the text processing tools to be applied to the compared segments. In addition, it details the modules included in the metric and the information they provide, together with examples of the information the user receives. Although VERTa is an MT evaluation metric, it differs from the rest in that during its development special emphasis was placed on analyzing the linguistic information it should provide to the user, thus going beyond a mere scoring of the translated segment and serving as a first qualitative guide to detect machine translation errors. Consequently, VERTa can be used for the learning, teaching and evaluation of English and Spanish as second and/or foreign languages, as well as to carry out research studies in this area.