B.Sc. (Computational) Linguistics - LIN-BS-070 - Winter 20/21

Seminar: Evaluation of NLP Systems

	Brielen Madureira
	Online, Wednesdays at 12:00
	04.11.2020 - 10.02.2021
	Moodle page, course EvalNLP id 25477
	madureiralasota at uni-potsdam dot de

The information here may still be changed or updated until Nov 2 without notice. After that, any modification will be discussed/announced in the Moodle forum.

Description

Baseline, F1 Score, black box, ablation, diagnostic, extrinsic/intrinsic, performance, annotation, metrics, human-based, test suite… Terms like these constantly show up in NLP papers, books and codes. What do they have in common? They are related to the evaluation process of systems. An adequate and fair evaluation is an essential step when building, analyzing and comparing models or algorithms. In this course, we will cover the main aspects of current Machine Learning evaluation methods and how the NLP community has been adapting them to the specific needs of different NLP tasks.

Goals

• Become aware of the importance of evaluation in NLP
• Get acquainted with the main evaluation methods, metrics and techniques
• Stay on top of how evaluation is being done for different NLP tasks

Format

Due to the pandemic, the course will happen 100% online. There will be weekly live meetings, but the main material (pre-recorded talks and exercises) will be available asynchronously, so that students with special time constraints can still follow the course.

The course will be divided into two parts. In the first half of the semester, we’ll learn about the evaluation methods in general machine learning. At the beginning of each week, I will post a pre-recorded short lecture with the week’s content. You can watch it anytime before your official slot, when we’ll meet to discuss your questions together and them use breakout sessions for you to discuss the week’s exercises in small groups. After the meeting, you have to submit your answers in Moodle.

In the second half of the semester, we will analyze evaluation in specific NLP tasks of your choice (e.g. dialogue, machine translation, text summarization, etc) with individual or small group presentations.

The course will be taught in English, but you can present and submit your assignments and the project in German if you prefer.

Please, check the guidelines and the schedule for more details. Note that this page is meant only to present the course, but we’ll use Moodle for everything else (submissions, announcements, question forums etc), so please enrol in the Moodle course page.

Background

Everyone is welcome to take this class, but you will profit more if you are already familiar with the main NLP tasks and methods. If you are in doubt, contact me :)

Requirements and grading policy

• active participation in the course (contributing to online group discussions and handing in exercises) [20% of final grade]
• presentation [40% of final grade]
• final project [40% of final grade]

You can get between 0 and 10 points for each item. To pass the course, you must achieve a performance of at least 70% on each of the items above separately, which includes abiding by the guidelines listed here.

Contact

Feel free to write me if you have any question or suggestion, or if you have any special needs (e.g. accessibility or name preference). I hope to see you soon!