Course Information

• Instructor: Ajit Rajwade
• Office: SIA-218, KReSIT Building
• Email: ajitvr@cse DOT iitb DOT ac DOT in
• Lecture Venue: CC 101 (New CSE Building, 1st floor)
• Lecture Timings: Slot 9, Monday and Thursday 3:30 to 5:00 pm
• Instructor Office Hours (in room SIA-218): Immediately after class
• Teaching Assistants: Jerin Geo James (jeringeo @ cse DOT iitb DOT ac DOT in)

Course Description

• This course will cover some state-of-the-art techniques for several applications in image processing and provide some rigorous theoretical foundation behind those.
• It will be of interest to students working in allied areas such as machine learning, statistics or signal processing, as well.
• Besides CSE and EE, this course will be of interest to students from various disciplines such as remote sensing (CSRE), earth sciences, computational biology, computational physics, etc.

• Denoising with different noise models, deblurring with different models of blur (motion/defocus)
• Tomographic reconstruction (in computed tomography machines, as well as in biology/virology)
• Reflection removal
• Image-based forensics
• Image compression
• Compressive reconstruction - including reconstruction of video, Magnetic resonance imaging (MRI) and hyperspectral images
• Some cutting-edge problems in image processing and machine learning: low rank matrix completion/recovery, phase retrieval, robust principal components analysis (RPCA) and compressive RPCA
• Time permitting, we will study the application of different types of convolution neural networks (CNNs) for solving inverse problems -- this is a relatively less explored facet of CNNs

Intended Audience and Pre-requisites

Textbooks

• "Natural Image Statistics", by Aapo Hyvarinen, Jarmo Hurri and Patrick Hoyer, Springer Verlag 2009, freely downloadable online
• "A Mathematical Introduction to Compressive Sensing", by Simon Foucart and Holger Rauhut, Birkhauser, 2013
• Statistical Learning with Sparsity: The Lasso and Generalizations", by Hastie, Tibshirani and Wainwright, CRC press (freely downloadable online)
• Various research papers on key topics will be circulated
  

Resources

• MATLAB at IITB
• MATLAB Tutorial: here or here
• MATLAB Image Processing Toolbox tutorial: here
• Matlab Tutorial
• The MathWorks - MATLAB Tutorial
• Matlab Primer
• On-line Matlab Help
• Writing Fast Matlab Code (pdf)
• Code Vectorization Guide
• Matlab Programmin Style Guidelines (pdf)

Grading scheme (tentative)

• Midsem 10%
• Cumulative Endsem 10%
• Class participation (questions in class, discussions with instructor or on moodle) 5%
• About 6-7 assignments involving programs, written problems and Google searches (yes you read that right -- but there's some (not so) fine print attached :-)) 50% (individually or in pairs)
• Course project with demo and viva 25% (individually or in pairs). You are expected to put in significant effort in the course project (in terms of reading literature as well as implementation and mathematics) and assignments.
• 100% attendance is expected - the only exception is documented emergencies or participation in conferences. Students with less than 80% attendance may get DX grade.
• Audit students must (1) write both exams and submit all assignments + project, and (2) have an aggregate score of at least 50%, for the AU grade.

Webpages of Previous Offerings

• Course webpage: Inaugural Offering
• Course webpage: Second Offering

Detailed Schedule

Date	Content of the Lecture	Assignments/Readings/Notes
3rd Jan (Thu)	Course overview	Slides (check moodle)
7th Jan (Mon)	Compressed Sensing Compressed sensing (CS): introduction and motivation Review of DFT and DCT, Review of JPEG Sparsity of natural images in transform bases Candes, Romberg, Tao: puzzling experiment; Basic optimization problem for CS involving the total variation Concept of incoherence between sensing matrix and representation basis	Slides (check moodle) E. Candes and M. Wakin, "Introduction to Compressive sampling", IEEE Signal Processing Magazine, 2008
10th Jan (Thu)	Shannon's sampling theorem and the Whittaker-Shannon Interpolation Formula Basic CS optimization problem: comments on the uniqueness of its solution Basic theorem of compressed sensing involving coherence (due to Candes, Romberg, Tao) Intuition behind role of incoherence in CS	Slides (check moodle) E. Candes and M. Wakin, "Introduction to Compressive sampling", IEEE Signal Processing Magazine, 2008
14th Jan (Mon)	Restricted isometry property Basic theorems of CS: reconstruction of compressible signals Compressed sensing under noise (theorem 3): choice of regularization parameter under different noise models The role of randomness Basis pursuit and its efficiency - Basis pursuit as a linear programming problem Intuitive explanation: benefits of L1 or L2 penalty for signal sparsity Toy examples of CS results	Slides (check moodle) E. Candes and M. Wakin, "Introduction to Compressive sampling", IEEE Signal Processing Magazine, 2008 J. Romberg, "Imaging via Compressive sampling", IEEE Signal Processing Magazine, 2008
17th Jan (Thu)	Sketch of a conventional camera Rice single pixel camera and its patchwise variant from Stanford Video version of the Rice SPC Conventional hyperspectral camera Architecture of CASSI: Coded aperture snapshot spectral imager	Slides (check moodle) Duarte et al, "Single-pixel imaging via Compressive sampling", IEEE Signal processing magazine, 2008 Kittle et al, "Multiframe image estimation for coded aperture snapshot spectral imagers", Applied Optics, 2010
21st Jan (Mon)	Architecture of CASSI: Coded aperture snapshot spectral imager Concept for color filter arrays (CFA), Comparison of CASSI with color image demosaicing Video compressive sensing using coded snapshots	Slides (check moodle) Kittle et al, "Multiframe image estimation for coded aperture snapshot spectral imagers", Applied Optics, 2010 Hitomi et al, "Video from single exposure coded snapshots", ICCV 2011
24th Jan (Thu)	Video compressive sensing using coded snapshots Compressed sensing in MRI: concept of different MRI trajectories CS algorithms: matching pursuit (MP) and orthogonal matching pursuit (OMP)	Slides (check moodle)
28th Jan (Mon)	CS algorithms: ISTA (iterative shrinkage/thresholding algorithm) Concept of mutual coherence and its relationship with the restricted isometry property/constant via Gershgorin's disc theorem Comparison between mutual coherence and RIC Theorem for performance bounds using mutual coherence	Slides (check moodle)
31st Jan (Thu)	RIC interpreted in terms of singular values, relationship between mutual coherence and RIC CS bounds based on mutual coherence (Theorem 5) CS bounds based on RIC of order s (as opposed to 2s): Theorem 6 Compressive classification: maximum likelihood classifier, matched filter, smashed filter, relation to RIP Sketch of proof of Theorem 3	Slides (check moodle) Candes, "The restricted isometry property and its implications for compressed sensing", CRM 2008 Davenport et al, "The smashed filter for compressive classification and target recognition", SPIE 2007
4th Feb (Mon)	Theorem 4: CS for piecewise constant signals CS with tight frames Clarification regarding RIP of Random Discrete Fourier Measurement Matrices Tomography Concept of tomography and tompgraphic projection/Radon transform Concept of backprojection and its limitations Generations of CT (computed tomography) machines	Slides (check moodle)
7th Feb (Thur)	Fourier slice theorem in tomography Concept of filtered backprojection and its relationship to (unfiltered) backprojection Tomography as a compressed sensing problem	Slides (check moodle)
9th Feb (Sat, extra lecture)	Tomography as a compressed sensing problem Tomography under unknown angles: motivation in cryo-EM, algorithms for 2D images Moment based approach - Helgason Ludwig consistency conditions (2D images) Concept of micrograph and particle picking in cryo-EM	Slides (check moodle)
11th Feb (Mon)	Moment-based algorithm (2D images) Ordering-based algorithm: nearest neighbor algorithm (2D images)	Slides (check moodle)
15th Feb (Thu)	Graph-laplacian algorithm for dimensionality reduction (due to Belkin and Niyogi); its application to tomography under unknown angles PCA-based algorithm for denoising of tomographic projections	Slides (check moodle)
21st Feb (Thu)	PCA-based algorithm for denoising of tomographic projections; derivation of a Wiener filter in the transform domain Tomography under unknown angles in 3D (i.e. 2D projection images): the concept of common lines, algorithm for finding common lines in spatial or Fourier domain Algorithm for finding unknown angles and underlying 3D structure from 2D projections under unknown angles - case of 3 projections only Orthogonal procrustes algorithm	Slides (check moodle)
4th March (Mon)	Distribution of midsem papers and discussion of solutions Algorithm for finding unknown angles and underlying 3D structure from 2D projections under unknown angles - case of 3 projections only Orthogonal procrustes algorithm Algorithm for finding unknown angles (based on semi-definite programming) and underlying 3D structure from 2D projections under unknown angles - case of N projections	Slides (check moodle)
7th March (Thurs)	Dictionary Learning Introduction to overcomplete dictionaries Concept of overcomplete dictionaries: sinusoids and spikes examples Issues of uniqueness and sparsity of representation in overcomplete dictionaries, dictionary learning as a generalization of K-means Principal Components Analysis: Eigenfaces; relationship between DCT and PCA	Slides (check moodle)
11th March (Mon)	A basic algorithm for simultaneously obtaining dictionary and sparse code - using projected gradient descent with adaptive step-size Method of Optimal Directions (MOD) Union of Orthonormal Bases KSVD method	Slides (check moodle) Aharon and Elad, K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, IEEE TSP 2006
14th March (Thurs)	KSVD method KSVD method for compression, denoising, inpainting	Slides (check moodle) Aharon and Elad, K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, IEEE TSP 2006
18th March (Mon)	Blind compressed sensing Compressive KSVD algorithm Requirement for blind compressed sensing: diversity of measurement matrices Fisher's linear discriminant: case of 2 classes	Slides (check moodle) Aharon and Elad, K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, IEEE TSP 2006 Anaraki and Hughes, Compressive KSVD, ICASSP 2014
25th March (Mon)	Fisher's linear discriminant: case of 2 classes Fisher's linear discriminant: case of more than 2 classes Limitations of FLD, introduction to mutual information for classification, concept of mutual information and its properties Mutual information for optimal transform learning for classification Kernel density estimation Optimization of quadratic mutual information	Slides (check moodle) Torkkola, "Feature Extraction by Non-Parametric Mutual Information Maximization", JMLR 2003
28th March (Thu)	Low Rank Matrix Recovery and Beyond Ubiquitousness of low rank matrices in image processing and machine learning: recommender systems, image patches assembled to form a matrix, distance matrices, applications in structrue from motion, applications in face recognition The problem of low rank matrix completion Informal statement of the basic theorem, concept of coherence of subspaces, sufficient conditions for successful recovery and pathological cases Formal statement of theorem of low rank matrix completion; extension to the case of noisy matrix completion Empirical results for low rank matrix completion Concept of low rank matrix recovery Basic theorems for low rank matrix recovery, concept of RIP of linear maps/operators, relation between low rank matrix recovery and low rank matrix completion	Slides (check moodle)
1st April (Mon)	Introduction to Robust Principal Components Analysis (PCA) Motivating examples for RPCA: background subtraction in videos, specularity and shadow removal from face images Basic theorem for RPCA; extension to the case when there is noise in observed matrix Basic theorem for RPCA with incompletely observed entries Singular Value Thresholding Algorithm (SVT) for matrix completion	Slides (check moodle)
4th April (Thurs)	Introduction to Robust Principal Components Analysis (PCA) Motivating examples for RPCA: background subtraction in videos, specularity and shadow removal from face images Basic theorem for RPCA; extension to the case when there is noise in observed matrix Basic theorem for RPCA with incompletely observed entries Singular Value Thresholding Algorithm (SVT) for matrix completion	Slides (check moodle)
8th April (Mon)	Inverse Problems using Deep Neural Networks Inverse problems from a Bayesian perspective; limitations of analytical models Deep neural networks for inverse problems: introduction Multi-layer perceptrons; Image denoising using MLPs Motivation for convolutional neural networks; their use for inverse problems Encoder-decoder CNNs Auto-encoders for super-resolution	Slides (check moodle) Paper on deep neural networks for inverse problems
11th April (Thurs)	Low Rank Matrix recovery and Beyond Compressive RPCA: Greedy Algorithm by Waters et al, applications in video and hyperspectral CS, and RPCA with missing entries Basic Theorem for Compressive RPCA Statistics of Natural Images Power law Correlation of pixel values in natural images Statistics of DCT coefficients of natural images: laplacian models and justification for the same Brief introduction to the Haar wavelet transform	Slides (check moodle)
15th April (Mon)	Brief introduction to the Haar wavelet transform Revision of basic Bayesian statistics, concept of MAP and MMSE, posterior probabilities with gaussian likelihood and Lapalcian and Gaussian priors Statistics of wavelet coefficients of natural images, use of dependencies between wavelet coefficients as priors for image denoising and in compression Compressed sensing based on statistical priors	Slides (check moodle)
18th April (Thu)	Statistics of natural image categories: natural, man-made,etc. Image and scene scale A semi-automated method for reflection removal using statistics of image gradients: Iteratively reweighted least squares algorithm Results of the IRLS algorithm for reflection removal, mixture of Laplacian prior for image gradients	Slides (check moodle)