Computational Ethics for NLP

CMU CS 11830, Spring 2019

T/Th 10:30-11:50am, POS 146

Yulia Tsvetkov (office hours: Tuesday 12-1pm, GHC 6405), ytsvetko@cs.cmu.edu
Alan W Black (office hours: Wednesday 12-1pm, GHC 5701), awb@cs.cmu.edu
TA: Anjalie Field (office hours: Thursday 3-4pm, GHC 6609), anjalief@cs.cmu.edu

HW 3: Abusive Language Online

Due 11:59pm, Thursday 4/4

Submission: Email your assignment to ethicalnlpcmu@gmail.com. Attach 3-4 separate files: your write-up (titled FirstName_LastName_hw3.pdf), your predictions over test.tsv titled FirstName_LastName_test.tsv, your improved predictions over test.tsv titled FirstName_LastName_advanced.tsv (if completed), and a zip/tar folder containing your code. Code will not be graded


Goals

As we have discussed in class, abusive language on online platforms has become a major concern in the past few years. The goals of this assignment are to (1) create a classifier for identifying offensive language (2) explore some of the challenges and ethical issues surrounding automated approaches to identifying abusive language


Overview

The data for this assignment is available here . Please note that the data contains offensive or sensitive content, including profanity and racial slurs.

In this homework, we provide a data set of tweets annotated for offensiveness. This data was taken from the 2019 SemEval task on offensive language detection. The file “offenseval-annotation.txt" provides an overview of the annotation scheme. The files "train.tsv" and "dev.tsv" both have the same format. The first column (text) contains the text of a tweet, the second column (label) contains an offensiveness label:

For tweets marked as OFF, the third column (category) specifies the type of offensive speech:

For tweets marked as NOT, the third column is nan

Finally, the file "test.tsv" contains only a set of tweets (i.e. only the text column)


Basic Requirements

Completing the basic requirements will earn a passing (B-range) grade

Offensive language identification

Categorization of offense types

Predictions

Write-up

Submit a 2-4 page report (ACL format) titled FirstName_LastName_hw3.pdf to ethicalnlpcmu[at]gmail.com. The report should include:


Advanced Analysis

Improve your preliminary classifier for either offensive language identification or categorization of offensive types (specifically aim to improve F1 score). Creative model architectures or feature crafting will receive full credit, even if they do not improve results. Some sources for inspiration include:

In your report, include a description of your model and results over dev.tsv. Additionally, use your improved classifier to predict results over test.tsv and place these predictions in a file titled FirstName_LastName_advanced.tsv using the appropriate column heading ("label" or "category")


Grading (100 points)


Implementation Tips