Statistical foundations of machine learning
Autumn 2005

Administrivia

Instructor

Soumen Chakrabarti

Teaching assistant

Kriti Puniyani,

Time and place

Slot 13, Mon+Thu 6:30--8pm, 301 SIT

Course newsgroup

Watch cse.cs705 (internal access) on jeeves.cse.iitb.ac.in for announcements.

Lecture calendar

Helpful for revising before exams.

Assignments

Internal link to upload solutions to assignments.

Evaluation

Please complete and submit the first student evaluation to the TA.

Credit students will need to write a midterm, a few peer-evaluated surprise quizzes and a final, and do a few assignments (typically using Scilab, WEKA, and perhaps Java).

Audit students have to write only the final exam and, to pass, their score must be above the bottom 20 percent of the class in finals only.

Grades are kept in a password-protected area.

Resources

Primary books

• The Elements of Statistical Learning by Hastie, Tibshirani, Friedman
• Pattern Classification, 2nd ed., by Duda, Hart and Stork, Wiley-Interscience
• Convex Optimization by Stephen Boyd and Lieven Vandenberghe
• Machine Learning by Tom Mitchell

Supplementary books

• An Introduction to Probability Theory by Feller
• All of Statistics by Larry Wasserman
• Principles of Data Mining by Hand, Mannilla, Smyth

Additional reading

• Andrew Moore's lecture slides
• Scribe notes on SVM, courtesy Markus Kirsten and Alban Scholer.
• Numerical Recipes in C, online book

Software and manuals

• used in class.
• Download Scilab, a free alternative to Matlab. Here is help to migrate from Matlab to Scilab.
• Or you can use R. Here is a manual.
• Internal link to download Matlab; we have limited license seats so try to migrate to Scilab or R.

Tentative syllabus

Prerequisites:

Basic vector algebra, calculus prob/stat, Linear and non-linear optimization

Bayesian learning:

Bayes optimal error, parametric models, parameter estimation and smoothing, EM

Regression and classification:

Loss functions, Linear regression and discriminants, Local regression and kernels, Logistic regression, Bias-variance tradeoff, Regularization, Ridge and Lasso

Support vector machines:

Variants of the max-margin objective, Optimization techniques, Structure prediction, Ranking, Ordinal regression, Transductive learning

Ensembles:

Exponential loss, Bagging, boosting, ECOC

Spectral methods:

Principal component analysis, Singular value decomposition, Graph partitioning

Factor analysis:

Non-negative matrix factorization, Information bottleneck, Co-clustering and cross-associations

Density estimation (tentative):

Kernel density methods, Basis function and wavelets

Graphical models (tentative):

Directed Bayesian belief networks, Markov random networks, probabilistic relational models

Eligibility

Mtech1 and DD4s who have not yet taken CS610 are the primary targets for this course. But it is open to all PG, DD and Btech4 students of all departments subject to department/facad approval. If you are CSE Btech3, ask for my consent by email while citing your CPI and grades in Paradigms and Prob/Stat.

This course should be reasonably simple if you have a little background in vector algebra, calculus, and probability. Some basic algorithms background may also help.

If you have already taken CS610, you will find quite some overlap. We are in a transition year: next year onwards, CS610 will leave out all CS705 material and specialize more on text, Web, and XML applications.

CS610 in Spring 2006 will not list CS705 as a prerequisite. However, you may be at a disadvantage if you cannot master on your own the Fall 2005 material.

Mtech2s who have taken CS610 in Spring 2005 are discouraged from taking the course on credit.

There will be slight (about 2--3 lectures at most) overlap with IT655.