B.Sc. (Computational) Linguistics - LIN-BS-070 - Winter 22/23

Seminar: Evaluation of NLP Systems

	Brielen Madureira
	Wednesdays at 12:00, room 2.14.0.32
	19.10.2022 - 08.02.2023
	Moodle page, course EvalNLP id 34082
	madureiralasota at uni-potsdam dot de

The information here may still be changed or updated until Oct 21 without notice. After that, any modification will be discussed/announced in the Moodle forum.

Description

Baseline, F1 Score, black box, ablation, diagnostic, extrinsic/intrinsic, performance, annotation, metrics, human-based, test suite, responsible NLP… Terms like these constantly show up in NLP papers, books and codes. What do they have in common? They are related to the evaluation process of systems. An adequate and fair evaluation is an essential step when building, analyzing and comparing models or algorithms. In this course, we will cover the main aspects of current Machine Learning evaluation methods and how the NLP community has been adapting them to the specific needs of different NLP tasks.

Goals

• Become aware of the importance of evaluation in NLP
• Get acquainted with the main evaluation methods, metrics and techniques
• Know how to use common metrics from Python libraries
• Stay on top of how evaluation is being done for different NLP tasks
• Think critically about ethical and responsible evaluation

Format

There will be weekly in person meetings for short lectures and group discussions. The material (slides, reading list and worksheets) will be available asynchronously, so that students with special time constraints can still follow the course. Notice that, although you may not come to the meetings, the group discussions are a core component in this course and the lectures will not be recorded.

The course will be divided into two parts. In the first half of the semester, we’ll learn about the evaluation methods in general machine learning. At the beginning of each week, I will post a reading list with the week’s content. You go over it anytime before our official slot and fill in the documentation. In our official slot, I’ll give a short lecture about the week’s topic, we’ll discuss your questions together and them you will discuss the week’s worksheet in small groups. After the meeting, you have to submit your answers on Moodle.

In the second half of the semester, we will analyze evaluation in specific NLP tasks of your choice (e.g. dialogue, machine translation, text summarization, etc) and dive deeper into topics on ethics in NLP with individual or small group presentations (depending on the number of participants) and guided discussions.

The course will be taught in English, but you can present, discuss and submit your assignments and the project in German if you prefer.

Please, check the guidelines and the schedule for more details. Note that this page is meant only to present the course, but we’ll use Moodle for everything else (submissions, announcements, question forums etc), so please enrol in the Moodle course page.

Background

Everyone is welcome to take this class, but you will profit more if you are already familiar with some NLP tasks and methods (e.g. CLT). Some knowledge of Python (e.g. PRS + ongoing PRO1) is needed for the programming assignments. If you are in doubt, contact me :)

Requirements and grading policy

To pass the course, you need to abide by the guidelines listed here and fulfill two ungraded requirements:

• active participation in the course (weekly documentation, weekly worksheets and programming assignments) [minimum of 60% of valid documentation submission, minimum of 60% of valid worksheet submissions + 2 mandatory assignments]
• presentation [at least 60% of the total points]

To pass the module, you need to hand in and pass a graded final project.

Contact

Feel free to write me if you have any question or suggestion, or if you have any special needs (e.g. accessibility or name preference). I hope to see you soon!