COL 775: Deep Learning
(Semester II, 2023-24)



General Information

Semester: Sem II, 2023-24.

Instructor: Parag Singla (email: parags AT cse.iitd.ac.in)

Class Timings/Venue:
  • Slot C. Tuesday, Wednesday, Friday: 8 am - 9 am. [Tentative: Any make-up classes will happen on Saturdays 8 am - 9 am].
  • Venue: LHC 526

Teaching Assistants:
To be Announced.


Registration Class List
Fill the following form if you are interested in registering for the course (and not yet registered) - only for CSE/ScAI/SIT students at this point.
Registration Form

Class List and TA Assignment: Coming soon!

Sign up for Piazza
Code: same as the course number (in small letters, with no spaces).

Pre-requisites

 A foundational course in AI or ML.

Announcements
  • [Jan 2, 2023]: First Class!

    Course Objective and Content

    Obective: This course is meant to be the first graduate level course in deep learning. Deep Learning is an emerging area of Machine Learning which has revolutionized the progress in the field during last few years with applications found in NLP, Vision and Speech to name a few domains. This course is intended to give a basic overview of the mathematical foundations of the field, and present a number of techniques/arhitectures which have become de-facto standard for a large number of applications in vision, NLP and other areas. We will also touch upon some of the recent advances in the field. become basis for more advanced ones. Without an implementation, no deep learning class can be complete. Students will get to implement some of the architectures on a GPU to test on large datasets.

    Content: Basics: Introduction. Multi-layered Perceptrons. Backpropagation. Regularization: L1-L2 Norms. Dropouts. Optimization: Challenges. Stochastic Gradient Descent. Advanced Optimization Algorithms. Batch and Layer Normalization. Convolutional Networks (CNNs). Pre-trained networks. Recurrent Architectures. Transformers. Generative Architectures. Graph Neural Networks. Deep Reinforcement Learning. Advanced Topics.

    Week-Wise Schedule (to be refined as we progress in the semester)

     NOTE: The exact list of topics below is tentative (until we are past that week). We will update it as we go through the lectures in each week. So, stay tuned!
    WeekTopic Book ChaptersSupplementary Material
    1 Introduction, Motivation
    2 Mulit-layered Perceptrons, Backpropagation Goodfellow et al. Chapter 6
    3 Regularization - L1/L2, Other Techniques Goodfellow et al. Chapter 7
    4 Optimization Techniques. Normalization Goodfellow et al. Chapter 8
    5 Deep Learning for Vision - Basic Models (CNNs) Goodfellow et al. Chapter 9
    6 Deep Learning for Vision (Architectures), Pre-Trained Models See list of papers
    7 Deep Learning for NLP - Basic Models (RNNs,LSTM) Goodfellow et al. Chapter 10
    8 Deep Learning for NLP (Attention), Transformers See list of papers
    9 Transformers - continued See list of papers
    10 Generative Models See class notes, list of papers
    11 Generative Models - continued See class notes, list of papers
    11 Graph Neural Networks
    12 Deep Reinforcement Learning
    14 Advanced Topics


    Class Notes/Videos (Date-Wise)

    Class Notes:

    Videos:

    Link to list papers covered (primarily in second half) [Last Offering of the course]

    Additional Reading/Papers

    Review Material

    Chapters 1 - 5, Goodfellow et al.

    Books/References

    1. Deep Learning. Ian Goodfellow, Yoshua Bengio and Aaron Courville. 2016.

    Assignment Submission Instructions
    1. You are free to discuss the problems with other students in the class. But the final solution/code that you produce should come through your individual efforts.
    2. Required code should be submitted using Moodle Page.
    3. Honor Code: Any cases of copying will be awarded a zero on the assignment. Additional penalities will be imposed based on the severity of copying. Any copying cases run the chances of being escalated to the Department/DISCO.
    4. Late policy: You are allowed a total of 3 buffer days (combined) for the all but the last programming assignment. There is no penalty if your submission stays within the limit of the 3 buffer days (total). For each additional day beyond the allowed 3 buffer days, you will lose 10% of the score for every late day in submission. Since last assignment will likely have a competitive part (to be communicated later), buffer days may not be applicable. We will update with more details as get closer to assignments.

    Practice Questions

    Assignments

    Grading Secheme (Tentative)

    Assignment 0 (tentative) 0%
    Assignment 1 (Part A, Part B) 15%
    Assignment 2 (Project) 20%
    Minor 25%
    Major 40%