Course Contents

Schedule

1. Administrivia.
   

Information Extraction

2. Topic: Open IE
   Reading: Open Information Extraction from Conjunctive Sentences.
   Additional Reading: Open Information Extraction: the Second Generation.
   Additional Reading (section 2): Open Information Extraction Systems and Downstream Applications.
   [Slides]


3. Topic: NELL
   Reading: Never-Ending Learning.
   Additional Reading: Toward an Architecture for Never-Ending Language Learning.
   [Slides]
4. Topic: Distant supervision using probabilistic graphical models
   Reading: Multi-instance Multi-label Learning for Relation Extraction.
   Background Reading: Distant supervision for relation extraction without labeled data.
   Background Reading: Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations.
   Additional Reading: Modeling Missing Data in Distant Supervision for Information Extraction.
   [Slides]
5. Topic: Neural distant supervision
   Reading: Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning.
   Background Reading: Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks.
   Additional Reading: Reinforcement Learning for Relation Classification from Noisy Data.
   [Slides]

Explainability in NLP

6. Topic: Attention vs. Explanation
   Reading: Attention and its (mis)interpretation.
   Additional Reading: Attention is not Explanation.
   Additional Reading: Attention is not not Explanation.
   Additional Reading: Is Attention Interpretable?.
   [Slides]

Pre-Trained Language Models

7. Topic: Transformer Architecture
   Reading: Attention is All You Need.
   [Slides]
8. Topic: BERT
   Reading: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
   Additional Reading: RoBERTa: A Robustly Optimized BERT Pretraining Approach.
   [Slides]
9. Topic: T5
   Reading: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
   [Slides]
10. Topic: GPT3
    Reading: Language Models are Few-Shot Learners.
    [Slides]
11. Topic: World Knowledge in Pre-trained Language Models
    Reading: Language Models as Knowledge Bases?.
    Additional Reading: BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA.
    [Slides]
12. Topic: Probing Pre-trained Language Models
    Reading: Visualizing and Measuring the Geometry of BERT.
    Additional Reading: BERT rediscovers the classical NLP pipeline.
    [Slides]

Machine Learning for NLP

13. Topic: Generative Adversarial Networks
    Reading: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.
    Background Reading: Adversarial Learning for Neural Dialogue Generation.
    [Slides]
14. Topic: Constrained Deep Learning
    Reading: A Primal-Dual Formulation for Deep Learning with Constraints.
    [Slides]
15. Topic: Tensor Factorization
    Reading: RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.
    Reading: Knowledge Base Completion: Baseline Strikes Back (Again).
    [Slides]

Question Answering

16. Topic: Open Domain QA
    Reading: REALM: Retrieval-Augmented Language Model Pre-Training.
    Background Reading: Reading Wikipedia to Answer Open-Domain Questions.
    Background Reading: R³: Reinforced Ranker-Reader for Open-Domain Question Answering.
    [Slides]
17. Topic: Reasoning for QA
    Reading: Neural Module Networks for Reasoning over Text.
    Background Reading: DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.
    Background Reading: Neural Module Networks.
    [Slides]
18. Topic: Multi-hop Reading Comprehension
    Reading: Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents.
    Background Reading: HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.
    Additional Reading: Hierarchical Graph Network for Multi-hop Question Answering.
    [Slides]
19. Topic: Semantic Parsing for QA
    Reading: Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing.
    Background Reading: Compositional semantic parsing on semi-structured tables.
    [Slides]
20. Topic: Knowledge-based QA
    Reading: Differentiable Reasoning over a Virtual Knowledge Base.
    Background Reading: PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text .
    [Slides]