Computational Ethics for NLP

HW 1: Corpus Exploration

Due 11:59pm, Thursday 2/7

Submission: Email your assignment to ethicalnlpcmu@gmail.com. Attach 2 separate files: your write-up (titled Lastname_Firstname.pdf) and a zip/tar folder containing your code. Code will not be graded

Goals

While many NLP tasks have standard data sets (e.g. SQuAD for Q&A), social-oriented tasks often involve using real-world data that has few annotations and unclear research questions. The goals of this assignment are: (1) Practice methods for exploring unstructured corpora (2) Conduct an analysis of language used to describe social groups (3) Consider the ethical implications of conducting analyses on sensitive social issues and unknown corpora

Overview

The data for this assignment is available here

In this homework you will explore how the language used to describe and report news about the LGBTQ community has changed over time by analyzing newspaper articles from 1986 to 2015.

You are given two corpora: a corpus of articles focused on the LGBTQ community (lgbt_news_corpus.csv) and a background corpus (background_news_corpus.csv). The primary data for analysis is in the “Text” column, and the remaining columns are metadata which you can optionally use.

Basic Requirements

Completing the basic requirements will earn a passing (B-range) grade

Analyze context: We provide a seed list (seed_list.txt) of words which are used to refer to members of the LGBTQ community. (Note: this list was not created by the organizers of this course. You can modify this list.) Analyze the context of the seed words with a window size of 3 and visualize analysis in the form of word clouds. Eg: If the sentence is: The pastoral guidelines remind priests that in dealing with homosexuals they must attentively adhere to the church’s magisterium, or traditional teachings, according to a church official. and the word to be considered from the seed list is homosexuals then you will analyze the words in, dealing, with, they, must, attentively. Consider how to subset the corpus to obtain interesting word clouds.

Analyze sentiment: Explore the sentiment used in the articles related to the LGBTQ community and compare with the background articles. We provide a lexicon of positive and negative sentiment words (sentiment_lexicon.csv). Analyze the intersection of these sentiment words with the articles that describe the LGBTQ community and the background topics, and report the differences that you find.

Write-up:

• Submit a 2-3 page report (ACL format). Because your analysis will include figures, there is no upper page limit, but please try to be concise
• Describe your methodology, including any decisions you made in data processing
• Present the results of your context, sentiment, and advanced analysis [if conducted]. Display your results clearly and point out any interesting trends.
• Briefly consider the ethical implications of your work (1-2 paragraphs). Consider questions like: who can be harmed by research on this topic? What could be concerns with your data set? You may include a discussion of one of the advanced analysis techniques even if you do not conduct the analysis. You are encouraged to cite external sources. For inspiration, refer to this blog post

Advanced Analysis

For full credit, complete an additional advanced analysis of this data set. Feel free to explore any further analysis directions. Here are a few references to give you ideas:

• Analyze the topics and words used across time by using Structural Topic Models
• Extract stereotypes that were associated with the LGBTQ community in the news across time. Suggested technique.
• Explore the use of biased words in corpora. Suggested technique, lexicons of epistemological bias.
• Extract metaphors associated with the LGBTQ individuals. Suggested technique.
• Extract aspect-based sentiment from the corpus, when the aspect is revealed to the LGBTQ seed terms. Relevant techniques.
• Analyze how meaning of the LGBTQ-related lexicons change over time. Suggested technique. If you are planning to explore this, contact us for a larger corpus of news.
• Further resources: additional sentiment lexicons are available here

Additional Tips

• You may use any programming language to conduct your analysis and you are welcome to use existing packages. Consider tools like nltk and gensim for text processing and tools like this package for visualization
• The files have some unusual formatting (welcome to real world messy data!). You may need to use codecs or a similar package to handle errors. Try: pandas.read_csv("lgbt_news_corpus.csv", encoding = "ISO-8859-1")

Grading (100 points)

• 20 points - Submitting assignment
• 40 points - Completing basic requirements
• 20 points - Write up is well-written, presents meaningful analysis, and contains all requested information
• 15 points - Advanced analysis
• 5 points - Discussion of ethical implications is particularly thoughtful or well-researched

Acknowledgements

Thank you Julia Mendelsohn (jmendels@stanford.edu) for sharing the data used in this assignment.