Video recordings and all course notes are all being made available on bodhi tree.
The course notes are also available from course page. The calendar page for the previous offering of this course is available here.

Date

Summary

-

Practice problem set: Please find here a growing list of practice problems. You can also refer to this wiki for definitions of several concepts relevant to this course, connections between those concepts through theorems and proofs of those theorems. It is very important that you attempt the homework problems that were posted at the end of (almost) each lecture. You can consider solving the following problems from PRML (Pattern Recognition and Machine Learning): 1.1 to 1.16, 1.20 to 1.32, 1.35 to 1.41, 2.1 to 2.8, 2.12, 2.13, 2.14 to 2.40, 3.2 to 3.10, 3.12 to 3.20, 4.2, 4.3, 4.4, 4.7, 4.9, 4.10, 4.12, 4.13, 4.14, 4.15 (harder), 4.16-4.21, 4.24, 5.1 - 5.9, 5.25 (hard), 5.27, 5.28, and following problems from Hastie et. al. : (A) 2.1, 3.2 , 3.5 , 3.6 , 3.7 , 3.12 , 3.19 , 3.21 , 3.23 , 3.27 , 3.28 , 3.29 , 3.30 , 4.2 , 4.3 , 4.4 , 4.5 , 4.6 , 4.7 , 4.8 , 5.16 , 6.2 , 6.4 , 6.6 , 6.9 , 6.11 , 8.1 , 8.2 , 8.5, 8.7, 10.5a, 10.8, 10.12, 11.2, 11.3, 11.4, 11.5, 11.7, 12.1, 12.2, 12.9, 12.10, 12.11 (harder), 13.1, 14.1, 14.2, 14.23 (harder)

Date	Topics
18-07-2016	Introduction to Machine Learning Unannotated Slides Annotated Slides Chapter 1 of Hastie et. al. Homework: Intuitively analyse the machine learning problem of handwritten digit recognition (slides 10 and 11) taking cues from the analysis on slides 8 and 9.
21-07-2016	Unsupervised learning vs. Supervised learning, Regression, Tutorial 0 Unannotated Slides Annotated Slides Refer to Sections 2.1 and 2.3.1 of Hastie et. al., Homework: Complete attempting problems from Tutorial 0
(Optional Revision Lecture) 23-07-2016, Contains solutions to most problems of Tutorial 0	Unannotated slided on probability revision and the corresponding annotated slides Unannotated slided on linear algebra revision and the corresponding annotated slides basic notes on linear algebra, basic notes on probability , advanced notes on probability. A good reference is Bayesian Reasoning and Machine Learning, by David barber, especially Chapter 1 and Appendix A.
25-07-2016	Basis functions/attributes, Least squared Regression and geometrical interpretation of its solution Unannotated Slides Annotated Slides Sections 2.2, 3.1 and 3.2 of Hastie et. al. Homework: Understand the concept of column space and the geometrical interpretation of the solution to the Least squared regression problem. What is polynomial regression on k indepenendent variables v1, v2.... vk and how would you achieve it using linear regression?
28-07-2016	Probabilistic Interpretation of Linear Regression Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 1 References: Section 3.1 (specifically until Section 3.1.3) of Pattern Recognition and Machine Learning by Christopher M. Bishop and Sections 2.2, 3.1 and 3.2 of Hastie et. al.
01-08-2016	Bayesian Estimation and Bayesian Linear Regression Unannotated Slides Annotated Slides Partial solutions to Tutorial 1 Homework: (Refer page 22 of the Unannotated Slides. Derive the posterior for the mean of a univariate Gaussian, given a 0 mean, fixed variance prior for the same. References: Sections 2.1, 2.1.1, 3.1.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop, Sections 3.4.3 of Hastie et. al.
04-08-2016	Conjugate Prior for (Multivariate) Gausssian, Posterior for (Multivariate) Gaussian, Bayesian Linear Regression Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 2. Also complete derivation of the MAP estimate for Bayesian Linear Regression (last slide of Annotated Slides) References: Sections 2.1, 2.1.1, 3.1.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop, Sections 3.4.3 of Hastie et. al.
08-08-2016	Concluding Bayesian Linear Regression, Ridge Regression, Lasso, Unannotated Slides Annotated Slides Solutions to Tutorial 2 Annotated Solutions to Tutorial 2 Homework: Complete the derivation on Slide 4 of the Annotated Slides References: Sections 3.3.1, 3.3.2, 2.3.3 and 2.3.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop, Sections 3.4.3 of Hastie et. al.
08-08-2016	Support Vector Regression and its Formulations, Optimization Basics Unannotated Slides Annotated Slides Solution to ML Homework of 8th August, courtesy Mohammad Shihabuddin Khan Tutorials 3 and 4 Homework: Complete solving Tutorials 3 and 4 References: Section 7.1.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop, Sections 3.8, 12.3.6 of Hastie et. al.
18-08-2016	Optimization Basics applied to regression formulations Unannotated Slides Annotated Slides Solutions to Tutorials 3 and 4 Homework:Explain the effect of lambda (page number 20) on positive definitesness of the Hessian in Ridge Regression Sections 3.2, 5.2.3 and Chapter 18 of Hastie et. al. and Section 4.1.4 of Optimization notes (assuming you are comfortable already until Section 4.1.3 from your basic calculus course).
22-08-2016	Discussion of solutions to Tutorials 3 and 4, Discussions on Positive Definiteness of Hessian, Gradient Descent, Line Search and Constrained Optimization Unannotated Slides Annotated Slides Solutions to Tutorials 3 and 4 Homework:What do you think should happen at w* if g1(w*)=0 on slide 16? Section 4.5.1 of Optimization notes
25-08-2016	Quiz 1, Discussion on Langrage Function, Necessary condition for constrained optimality, Geometric interpretation, intution behind the KKT necessary conditions Unannotated Slides Annotated Slides Quiz 1 Solutions to Quiz 1 Homework:How do you think the necessary conditions for constrained optimality will look like for equality constraints? How will these conditions look like with multiple inequality and equality constraints? Section 4.4.1 of Optimization notes
29-08-2016	Discussion on Karush Kuhn Tucker Conditions, Langrage Dual, Dual Optimization Problem, KKT Conditions on SVR, Concept of Support Vectors Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 5 Reference: Sections 3.4.2, 3.4.3, 3.8, 12.3.6 of Hastie et. al. and Sections 4.4.1, 4.4.2 of Optimization notes
01-09-2016	Dual of SVR, Kernel, Positive Definite/Mercer Kernels, Solutions to Tutorial 5, Unannotated Slides Annotated Slides Solutions to Tutorial 5 Annodated solutions to Tutorial 5 Reference: Sections 12.3.6, 5.8.1, 2.8.1 and 2.8.2 of Hastie et. al. and this paper on kernel ridge regression Section 4.5.1 of Optimization notes
15-09-2016	Non-Parametric Regression, Algorithms for Optimizing SVR and Lasso Unannotated Slides Annotated Slides Midsem Solutions to Midsem Reference: Sections 5.8.1, 2.8.1 and 2.8.2 of Hastie et. al.
20-09-2016	Classification, Binary Classification using Perceptron, Rationale behind perceptron update, gradient descent and stochastic gradient descent for perceptron updates Unannotated Slides Annotated Slides Solutions to Midsem Annotated solutions to Midsem Reference: Sections 4.5.1 of Hastie et. al. Homework: Try and prove the convergence of the perceptron update rule on the last slide of the annotated slides .
22-09-2016	Convergence of Perceptron Update Algorithm, Logistic Regresion (Sigmoid Perceptron), From Maximum likelihood to Minimum Cross Entropy Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 6. Derive the cross entropy minimization formulation for Logistic Regression Parameter estimation from the maximum likelihood formulation Reference: Sections 4.5.1, 4.4 and 11.4 of Hastie et. al.
26-09-2016	Logistic Regresion (Sigmoid Perceptron), Minimum Penalized Cross Entropy and its Bayesian Interpretation, Gradient Descent update and other optimization algorithms, Multiclass Logistic Regression Unannotated Slides Annotated Slides References: Sections 2.8.1, 4.4, 8.2, 8.3 of Hastie et. al. and reading on Bayesian Logistic Regression.
29-09-2016	Multiclass Logistic Regression, Kernelized Logistic Regression for Non-linear seperation, Conditional Random Fields, Non-linear seperation through Neural Networks Unannotated Slides Annotated Slides Solutions to Tutorial 6 Annotated Solutions to Tutorial 6 Homework: Complete attempting problems from Tutorial 7. References: Sections 8.2, 8.3, 11.4 (Neural Network training) of Hastie et. al., Article by Trevor Hastie on Kernel Logistic Regression, CRF extension to Logistic Regression at tutorial at CIKM 2008 (Sections 1.4, 3.1 and 4.1) and on slides 7 to 11 here, Extra Reading(s): Structured Prediction through Graphical Models.
03-10-2016	Non-linear seperation through Neural Networks, Notations, Backpropagation Algorithm: Basic sketch and motivation Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 7. References: Sections 8.2, 8.3, 11.4 (Neural Network training) of Hastie et. al., Chapter 6 (esp Sections 6.1, 6.2.2, 6.3, 6.5 and Algo 6.5) of the online book on Deep Learening
06-10-2016	Backpropagation Algorithm in Details, Reasons for recent advances in deep learning, introduction to the "Lego" world of Deep learning (through convolutional neural networks) Unannotated Slides Annotated Slides References: Chapter 8 (esp Section 8.3) of the online book on Deep Learening, Sections 8.2, 8.3, 11.4 (Neural Network training) of Hastie et. al.,
10-10-2016	"Lego" world of Deep learning, Convolutional Neural networks Unannotated Slides Annotated Slides Solutions to Tutorial 7 References: Chapter 9 (esp Section 9.1, 9.2) of the online book on Deep Learening, Section 11.7 of Hastie et. al.,
13-10-2016	Convolutional Neural networks with more examples and illustrations, Recurrent Neural networks with examples and illustrations Unannotated Slides Annotated Slides Homework: Complete attempting problems from Tutorial 8. References: Chapter 9 (esp Sections 9.3, 9.4, 9.5) of the online book on Deep Learning, Chapter 10 (esp Section 10.1, 10.2) of the online book on Deep Learening
17-10-2016	Long Short Term Memory (LSTM), Connectionist Temporal Classification, Decision Trees Unannotated Slides Annotated Slides Solutions to tutorial 7. Annotated Solutions to tutorial 7. Solutions to Tutorial 8. References: Chapter 10 (esp Section 10.10) of the online book on Deep Learening, Section 9.2.3 (Decision Trees) of Hastie et. al., Homework: Build decision trees for the problems on page 31 of these class slides
20-10-2016	Decision Trees, Impurity measures, Minimum Description Length, Post-pruning Unannotated Slides Annotated Slides Solutions to Quiz 2. References: Section 9.2.3 (Decision Trees) of Hastie et. al.,
24-10-2016	Impurity Measures and Feature Selection, Ensemble Learning, Bagging and Boosting, Adaboost Algorithm, Support Vector Classification Unannotated Slides Annotated Slides Tutorial 9 along with solutions. Tutorial 9 annotated along with solutions. References: Sections 8.7, 10.1-10.4 of Hastie et. al., Sections 14.2, 14.3, 14.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop Extra Readings: Additive Logistic Regression: a Statistical View of Boosting, Consistency of Random Forests and Other Averaging Classifiers, Overview of the Probably Approximately Correct (PAC) Learning Framework, Section 7.3 of Hastie et. al., Section 3.2 of Pattern Recognition and Machine Learning by Christopher M. Bishop
27-10-2016	Support Vector Classification, Hinge Loss, Representer Theorem, SVC Dual, (Word) Embeddings Unannotated Slides (Part a) Unannotated Slides (Part b) Annotated Slides (Part a) Annotated Slides (Part b) References: Sections 8.7, 10.1-10.4 of Hastie et. al., Sections 14.2, 14.3, 14.4 of Pattern Recognition and Machine Learning by Christopher M. Bishop Extra Readings: Additive Logistic Regression: a Statistical View of Boosting, Consistency of Random Forests and Other Averaging Classifiers, Overview of the Probably Approximately Correct (PAC) Learning Framework, Section 7.3 of Hastie et. al., Section 3.2 of Pattern Recognition and Machine Learning by Christopher M. Bishop
03-11-2016	Generative Models, Gaussian Discriminant Analysis, Unsupervised Learning and EM Algorith, Gaussian Mixture Model, K-Means and variants (remaining slides are optional: on Hierarchical Clustering and Distance measures, Multinomial Naive Bayes and MLE/MAP Estimation for the same, Multilabel classification) Unannotated Slides Annotated Slides See Tutorial 10 See Solutions to Tutorial 10 See Annotated solutions to Tutorial 10 References: Sections 8.5.1 (EM Algorithm for Mixture of Gaussians), 8.5.2 (General EM Algorithm), 13.2.1 and 14.3.6 and 14.3.7 (K-means clustering), 14.3.2 and 14.3.3 (Distance/dissimilarity measures), 14.3.10 (K-mediods algorithm), 14.3.12 (Extra: Hierarchical Clustering) of Hastie et. al., Section 13 (Clustering) and Section 12.2 (EM Algo for clustering) of previous offering's class notes and Section 7.8 (more reading on unsupervised learning and EM algorithm) of notes on learning and inferencing in probabilistic models References: Sections 4.3 (Linear and Quadratic Discriminant Analysis), 4.3.1 (regularization for Gaussian Disriminant Analysis), 12.4 (Optional: Generalized Linear Discriminant), 13.2.3 (Mixture of Gaussians), 14.3.7 (EM for Mixture of Gaussians as soft-clustering) of Hastie et. al., Section 11 (Gaussian Discriminant Analysis) and Section 12 (Mixture of Gaussians, EM and clustering) of previous offering's class notes and Section 7.8 (more reading on unsupervised learning and EM algorithm) of notes on learning and inferencing in probabilistic models