Introduction to Machine Learning (ELL784)


General Information

No one shall be permitted to audit the course. People are welcome to sit through it, however. The course is open to all suitably inclined M.Tech, M.S.(R) and Ph.D. students of all disciplines. This course is not open to B.Tech and Dual Degree students, who are supposed to opt for ELL409 (Mahine Intelligence and Learning). This is a Departmental Elective (DE), one of the `essential electives' for the Cognitive and Intelligent Systems (CIS) stream of the Computer Technology Group, Department of Electrical Engineering. A general note for all EE Machine Learning courses: students will be permitted to take only one out of the following courses: ELL409 (Machine Intelligence and Learning), and the two CSE Machine Learning courses: COL341 Machine Learning and COL774 Machine Learning.

Credits: 3 (LTP: 3-0-0) [Slot C]

Schedule for Classes:

Tuesday
08:00 - 09:00
IIA-101 (Bharti Building)
Wednesday
08:00 - 09:00
IIA-101 (Bharti Building)
Friday
08:00 - 09:00
IIA-101 (Bharti Building)

Schedule for Examinations:

Minor I: 05 February 2019 (Tuesday), 11:00 am - 12:00 pm, LH-310
Minor II: 26 March 2019 (Tuesday), 11:00 am - 12:00 pm, LH-310
Major: 28 April 2019 (Sunday), 10:30 am - 12:30 pm, LH-310

Teaching Assistants: 

Sakshi Agrawal
Hareesh Kumawat

Books, Papers and other Documentation

Textbook:

Reference Books:

Papers:

Some Interesting Web Links:


Lecture Schedule, Links to Material

Please see the link to the II Sem (Spring) 2017-2018 offering of this course, for an idea of the approximate structure of the course.
S.No.
Topics
Lectures
Instructor
References
0
Introduction to Machine Learning
01-01
SDR
Flavours of Machine Learning: Unsupervised, Supervised, Reinforcement, Hybrid models. Decision Boundaries: crisp, and non-crisp, optimisation problems. Examples of unsupervised learning.
01 Jan (Tue) {lecture#01}
SDR
1
Unsupervised Learning:
K-Means, Gaussian Mixture Models, EM
02-06
SDR
[Bishop Chap.9], [Do: Gaussians], [Do: More on Gaussians], [Ng: K-Means], [Ng: GMM], [Ng: EM], [Smyth: EM]
The K-Means algorithm: two flavours. Algorithms: history, flavours. A mathematical formulation of the K-Means algorithm. The Objective function to minimise. The basic K-Means algorithm, computation complexity issues: each step, overall. Limitations of K-Means.
02 Jan (Wed) {lecture#02}
SDR
K-Means: Alternate formulation with a distance threshold. Gaussian Mixture Models. The Bayes rule, and Responsibilities. Maximum Likelihood Estimation. Parameter estimation for a mixture of Gaussians, starting with a simple 1-D single Gaussian case.
04 Jan (Fri) {lecture#03}
SDR
{Early start: 7:30am-9:00am class}
--- No class ---
08 Jan (Tue) {lecture#xx}
---
--- No class ---
09 Jan (Wed) {lecture#xx}
---
--- No class ---
11 Jan (Fri) {lecture#xx}
---
ML-Estimation: the simple case of one 1-D Gaussian, to the general case of K D-dimensional Gaussians.
15 Jan (Tue) {lecture#04}
SDR
{Early start: 7:30am-9:00am class}
The general case of K D-dimensional Gaussians. Getting stuck, using Lagrange Multipliers. The EM Algorithm for Gaussian Mixtures.
Application: Assignment 1: The Stauffer and Grimson Adaptive Background Subtraction Algorithm. An introduction to the basic set of interesting heuristics!
16 Jan (Wed) {lecture#05}
SDR
{Early start: 7:30am-9:00am class}
[CVPR'99], [PAMI'00]
The Stauffer and Grimson algorithm (contd)
18 Jan (Fri) {lecture#06}
SDR
{Early start: 7:30am-9:00am class}
2
Unsupervised Learning: EigenAnalysis:
PCA, LDA and Subspaces
07-09
SDR
[Ng: PCA], [Ng: ICA], [Burges: Dimension Reduction], [Bishop Chap.12]
Introduction to Eigenvalues and Eigenvectors. Properties of Eigenvalues and Eigenvectors
22 Jan (Tue) {lecture#07}
SDR
{Early start: 7:30am-9:00am class}
Gram-Schmidt Orthogonalisation, other properties. The KL Transform
23 Jan (Wed) {lecture#08}
SDR
{Early start: 7:30am-9:00am class}
The SVD and its properties
25 Jan (Fri) {lecture#09}
SDR
{Early start: 7:30am-9:00am class}
3
Linear Models for Regression, Classification
10-14
SDR
[Bishop Chap.3], [Bishop Chap.4], [Ng: Supervised, Discriminant Analysis], [Ng: Generative],
General introduction to Regression and Classification. Linearity and restricted non-linearity. Maximum Likelihood.
29 Jan (Tue) {lecture#10}
SDR
{Early start: 7:30am-9:00am class}
Maximum Likelihood and Least Squares. The Moore-Penrose Pseudo-inverse.
30 Jan (Wed) {lecture#11}
SDR
--- No class ---
01 Feb (Fri) {lecture#xx}
SDR
---
Minor I
05 Feb (Tue), LH-310, 11am-12pm
---
---
Regularised Least Squares.
Classification (an introduction).
08 Feb (Fri) {lecture#12}
SDR
OpenCV Introduction
08 Feb (Fri) {lecture#xx}
SC
(EE Committee Room, 03:00pm-04:00pm)
Linear Discriminant Functions: 2 classes, and K classes. Fisher's Linear Discriminant.
12 Feb (Tue) {lecture#13}
SDR
Fisher's Linear Discriminant (contd.)
13 Feb (Wed) {lecture#14}
SDR
4
SVMs
15-28
SDR
[Bishop Chap.7], [Alex: SVMs], [Ng: SVMs], [Burges: SVMs], [Bishop Chap.6], [Khardon: Kernels]
Introduction to SVMs
19 Feb (Tue) {lecture#15}
SDR
The basic optimisation problem: margin maximisation
20 Feb (Wed) {lecture#16}
SDR
Optimisation criteria. Getting the physical significance of the y = +1 and y = -1 lines. The two `golden' regions for the 2-class pefectly separable case. The generalised canonical representation in terms of one inequation.
22 Feb (Fri) {lecture#17}
SDR
The basic SVM optimisation
26 Feb (Tue) {lecture#18}
SDR
The basic SVM optimisation: the primal and the dual problems.
An illustration of the kernel trick
27 Feb (Wed) {lecture#19}
SDR
Lagrange Multipliers and the KKT Conditions
01 Mar (Fri) {lecture#20}
SDR
Lagrange Multipliers and the KKT Conditions (contd).
The Soft-Margin SVM.
12 Mar (Tue) {lecture#21}
SDR
The Soft-Margin SVM (contd).
13 Mar (Wed) {lecture#22}
SDR
The Soft-Margin SVM (contd).
15 Mar (Fri) {lecture#23}
SDR
The Soft-Margin SVM (contd).: A common inequation, the primal problem, getting the KKT conditions for the soft-margin problem.
19 Mar (Tue) {lecture#24}
SDR
Recap: Lagrange Multipliers and the KKT Conditions.
The soft-margin SVM (contd).
20 Mar (Wed) {lecture#25}
SDR
{Early start: 7:30am-9:00am class}
Introduction to Kernels.
22 Mar (Fri) {lecture#26}
SDR
---
Minor II
26 Mar (Tue), LH-310, 11am-12pm
---
---
Kernels in Regression
29 Mar (Fri) {lecture#27}
SDR
{Early start: 7:30am-9:00am class}
--- No class ---
30 Mar (Sat) {lecture#xx}
---
Kernel Functions: properties, construction
02 Apr (Tue) {lecture#28}
SDR
5
Neural Networks
33-40
SDR
[Bishop Chap.5], [Alex: NNRep], [Alex: NNLearn]
Introduction to Neural Networks: the Multi-Layer Perceptron: Conventions, restricted non-linearity
03 Apr (Wed) {lecture#29}
SDR
Basic Perceptron, Optimisation
05 Apr (Fri) {lecture#30}
SDR
Optimisation: Estimating network parameters for a regression problem
09 Apr (Tue) {lecture#31}
SDR
Optimisation: Estimating network parameters on a classification problem. The Binomial distribution, the Bernoulli distribution
Physical Significance of the NN solution vis-a-vis the linear method done before
10 Apr (Wed) {lecture#32}
SDR
Basic Optimisation basics: local quadratic approximation, geometric interpretation, computing the gradient
12 Apr (Fri) {lecture#33}
SDR
The Backpropagation Algorithm
16 Apr (Tue) {lecture#34}
SDR
Backpropagation vs numerical evaluation of the gradient
Invariance issues: linear transformation of the input and outputs
4 Basic Invariance-handling techniques
19 Apr (Fri) {lecture#35}
SDR
Some details on Tangent Propagation, CNNs
23 Apr (Tue) {lecture#36}
SDR
24 Apr (Wed) {lecture#37}
SDR
26 Apr (Fri) {lecture#38}
SDR
---
Major
28 Apr (Sun), LH-310, 10:30am-12:30pm
---
---
5
Feature Selection
xx-xx
SDR
6
Logistic Regression
28-29
SDR
[Alex: LogReg]
xx
Mathematical Basics for Machine Learning
xx-xx
xx
[Burges: Math for ML], [Burges: Math Slides], [Do, Kolter: Linear Algebra Notes],

[Internal Link: IIT Delhi]

The above list is (obviously!) not exhaustive. Other reference material will be announced in the class. The Web has a vast storehouse of tutorial material on AI, Machine Learning, and other related areas.



Assignments

... A combination of theoretical work as well as programming work.
Both will be scrutinized in detail for original work and thoroughness.
For programming assignments, there will be credit for good coding.
Sphagetti coding will be penalized.
Program correctness or good programming alone will not fetch you full credit ... also required are results of extensive experimentation with varying various program parameters, and explaining the results thus obtained.
Assignments will have to be submitted on or before the due date and time.
Late submissions will not be considered at all.
Unfair means will be result in assigning as marks, the number said to have been discovered by the ancient Indians, to both parties (un)concerned.
Assignment 1
Assignment 2
Assignment 3

Examinations and Grading Information

The marks distribution is as follows (out of a total of 100):
Minor I
25
Minor II
25
Assignments
25
Major
25
Grand Total
100

ELL784 Evaluation: Programming Assignment Groups, Assignment/Examination-related Information [Internal Link: IIT Delhi]

Attendance Requirements:

As per Institute rules for IIT Delhi students: a minimum of 75% (i.e., a maximum of 11 absents permitted), else one grade less.
Illness policy: illness to be certified by a registered medical practioner.
Attendance in Examinations is Compulsory.

ELL784 Attendance Records (on the moodle page)


Course Feedback

Link to Course Feedback Form

Sumantra Dutta Roy, Department of Electrical Engineering, IIT Delhi, Hauz Khas,
New Delhi - 110 016, INDIA. sumantra@ee.iitd.ac.in