Aufbauphase: Methoden der Computer- linguistik

University of Potsdam

B.Sc. (Computational) Linguistics - LIN-BS-070 - Winter 20/21

Seminar: Evaluation of NLP Systems

:woman_teacher: Brielen Madureira
:date: Online, Wednesdays at 12:00
:hourglass: 04.11.2020 - 10.02.2021
:computer: Moodle page, course EvalNLP id 25477
:e-mail: madureiralasota at uni-potsdam dot de

:warning: The information here may still be changed or updated until Nov 2 without notice. After that, any modification will be discussed/announced in the Moodle forum.


Baseline, F1 Score, black box, ablation, diagnostic, extrinsic/intrinsic, performance, annotation, metrics, human-based, test suite… Terms like these constantly show up in NLP papers, books and codes. What do they have in common? They are related to the evaluation process of systems. An adequate and fair evaluation is an essential step when building, analyzing and comparing models or algorithms. In this course, we will cover the main aspects of current Machine Learning evaluation methods and how the NLP community has been adapting them to the specific needs of different NLP tasks.



Due to the pandemic, the course will happen 100% online. There will be weekly live meetings, but the main material (pre-recorded talks and exercises) will be available asynchronously, so that students with special time constraints can still follow the course.

The course will be divided into two parts. In the first half of the semester, we’ll learn about the evaluation methods in general machine learning. At the beginning of each week, I will post a pre-recorded short lecture with the week’s content. You can watch it anytime before your official slot, when we’ll meet to discuss your questions together and them use breakout sessions for you to discuss the week’s exercises in small groups. After the meeting, you have to submit your answers in Moodle.

In the second half of the semester, we will analyze evaluation in specific NLP tasks of your choice (e.g. dialogue, machine translation, text summarization, etc) with individual or small group presentations.

The course will be taught in English, but you can present and submit your assignments and the project in German if you prefer.

Please, check the guidelines and the schedule for more details. Note that this page is meant only to present the course, but we’ll use Moodle for everything else (submissions, announcements, question forums etc), so please enrol in the Moodle course page.


Everyone is welcome to take this class, but you will profit more if you are already familiar with the main NLP tasks and methods. If you are in doubt, contact me :)

Requirements and grading policy

You can get between 0 and 10 points for each item. To pass the course, you must achieve a performance of at least 70% on each of the items above separately, which includes abiding by the guidelines listed here.


Feel free to write me if you have any question or suggestion, or if you have any special needs (e.g. accessibility or name preference). I hope to see you soon! :wink: