CS 754 - Advanced Image Processing



Course Information

Course Description

Besides being just two- or three-dimensional arrays containing numbers (pixel intensity values), the images that are seen and acquired in everyday life have a number of very interesting properties. For example, small patches from different spatial regions of an image are quite similar to each other. Another example is that these images when transformed by certain kinds of matrix operations (eg: wavelet or discrete cosine transform) produce coefficients that are very sparse (i.e. most of them are zero or close to zero in value). These properties are a key to developing efficient and accurate algorithms for a variety of applications such as image noise removal, blur removal, image separation, compression and even image-based forensics. Not just that, these properties have also inspired the design of new sensors to acquire images much faster (a technology called compressed sensing) which is really crucial in tasks involving video or some types of images in medicine (such as magnetic resonance imaging - MRI). This course will systematically study several such techniques motivated by these interesting properties of natural images. The course will explore some key theory (including the proofs of some truly beautiful theorems), algorithms, as well as many applications. It will expose students to a broad range of modern, state-of-the-art techniques in image processing. A more detailed (tentative) syllabus can be accessed here.

Need for the course Image Processing Applications

The course will cover principled techniques that can be applied to some interesting image processing problems:

Intended Audience and Pre-requisites

Intended for 3rd, 4th year B Tech students, DDP students, and all PG students (MTech/PhD) from CSE/EE/EP/CSRE. You must have taken CS 663 or EE 610 or CS 725, otherwise you must discuss with me whether you have suitable background for this course.

Textbooks

Resources


Grading scheme (tentative)


Last year's course webpage

Date

Content of the Lecture

Assignments/Readings/Notes

4th Jan
  • Course overview
Slides (check moodle)
8th Jan Compressed Sensing
  • Compressed sensing (CS): introduction and motivation
  • Review of DFT and DCT, Review of JPEG, Review of the Sampling theorem and its limitations
  • Sparsity of natural images in transform bases
  • Candes, Romberg, Tao: puzzling experiment
  • Basic optimization problem for CS
  • Slides (check moodle)
  • Research papers (check moodle)
  • Candes and wakin, "Introduction to Compressive sampling", IEEE Signal Processing Magazine, 2008
11th Jan
  • Three key theorems for compressed sensing and comments on the theorems
  • Concept of incoherence between sensing matrix and representation basis
  • Basic CS optimization problem: comments on the uniqueness of its solution
  • Concept of restricted isometry property
  • Slides (check moodle)
  • Research papers (check moodle)
  • Candes and wakin, "Introduction to Compressive sampling", IEEE Signal Processing Magazine, 2008
15th Jan
  • Compressed sensing under noise (theorem 3): choice of regularization parameter under different noise models
  • The role of randomness
  • Basis pursuit and its efficiency - Basis pursuit as a linear programming problem
  • Intuitive explanation: benefits of L1 or L2 penalty for signal sparsity
  • Toy examples of CS results
  • Rice single pixel camera
  • Slides (check moodle)
  • Research papers (check moodle)
  • Candes and wakin, "Introduction to Compressive Sampling", IEEE Signal Processing Magazine, 2008
  • Romberg,"Imaging via Compressive Sampling", IEEE Signal processing Magazine, 2008
18th Jan
  • Rice single pixel camera
  • Block-wise single pixel camera by El-Gamal
  • Video version of the Rice SPC
  • The CASSI architecture for compressed sensing of hyperspectral images
  • Slides (check moodle)
  • Research papers (check moodle)
  • Duarte et al, "Single-pixel imaging via Compressive sampling", IEEE Signal processing magazine
  • Kittle et al, "Multiframe image estimation for coded aperture snapshot spectral imagers", Applied Optics
22nd Jan
  • Discussion on compressive architectures: video version of Rice SPC, CASSI
  • Color image acquisition via color filter arrays and its relationship with CASSI acqusition
  • Video compressive sensing using coded snapshots
  • Slides (check moodle)
  • Research papers (check moodle)
  • Hitomi et al, "Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary", ICCV 2011
25th Jan
  • Video compressive sensing using coded snapshots
  • CS in Magnetic Resonance Imaging (MRI)
  • CS Theory: Reconstruction of piecewise flat signals
  • Concept of mutual coherence and its comparison to RIP/RIC: Theorem 5
  • Slides (check moodle)
  • Research papers (check moodle)
  • Lustig et al, "Compressed Sensing MRI", IEEE Signal Processing Magazine, 2008
29th Jan
  • CS Algorithms: Matching pursuit and Orthogonal Matching pursuit
  • Iterative Shrinkage and Thresholding Algorithm (ISTA)
  • Slides (check moodle)
  • Research papers (check moodle)
1st Feb
  • Compressed Sensing Theorem 3: sketch of the proof - tube constraint, cone constraint, various vector inequalities
  • Concept of overcomplete dictionaries
  • Concept of design of CS matrices my minimizing mutual coherence
  • Compressive classification
  • Slides (check moodle)
  • Research papers (check moodle)
  • Candes, "The restricted isometry property and its implications for compressed sensing", Comptes Rendus de Mathematiques, 2008
5th Feb
  • Compressive classification: maximum likelihood classifier, matched filter, smashed filter, relation to RIP
  • Concept of design of CS matrices my minimizing mutual coherence: method of Duarte-Carvajalino and Sapiro
Tomography
  • Concept of tomography and tompgraphic projection/Radon transform
  • Concept of backprojection and its limitations
  • Generations of CT (computed tomography) machines
  • Fourier slice theorem in tomography
  • Slides (check moodle)
  • Research papers (check moodle)
  • Davenport et al, "The smashed filter for compressive classification and target recognition", SPIE, 2007
8th Feb
  • Fourier slice theorem in tomography
  • Concept of filtered backprojection
  • Tomography as a compressed sensing problem
  • Slides (check moodle)
12th Feb
  • Tomography as a compressed sensing problem
  • Tomography under unknown angles: motivation, algorithms for 2D images
  • Moment based approach - Helgason Ludwig consistency conditions (2D images)
  • Moment-based algorithm (2D images)
  • Slides (check moodle)
15th Feb
  • Ordering-based algorithm: nearest neighbor algorithm (2D images)
  • Approaches based on dimensionality reduction: Graph Laplacian algorithm for dimensionality reduction (2D images)
  • Issue of unknown shifts
  • Slides (check moodle)
19th Feb
  • PCA-based algorithm for denoising tomographic projections
  • Tomography under unknown angles for 3D images - common lines constraint, detailed algorithm
  • Use of the SVD in the aforementioned algorithm, orthogonal Procrustes problem
  • Slides (check moodle)
19th Feb
  • Tompgraphy under unknown angles for 3D images - common lines constraint, detailed algorithm
  • Use of the SVD in the aforementioned algorithm, orthogonal Procrustes problem
  • Discussion session for midsem
  • Slides (check moodle)
2nd March
  • Distribution of midsem papers and discussion of questions
Dictionary Learning
  • Introduction to basics of dictionary learning
  • Principal components analysis (PCA)
  • Eigenfaces algorithms - efficient variant, eigenvalue decay for face images
  • Slides (check moodle)
5th March
  • Eigenfaces algorithms - efficient variant, eigenvalue decay for face images
  • Derivation for PCA - as an algorithm to infer an orthonormal matrix that minimizes reconstruction error, and hence maximizes variance
  • Extension of this derivation to handle multiple directions
  • Robust versions of PCA (eg: L1 PCA)
  • Applications of PCA in image compression
  • Slides (check moodle)
7th March (extra lecture)
  • Eigenfaces algorithms - efficient variant, eigenvalue decay for face images
  • Derivation for PCA - as an algorithm to infer an orthonormal matrix that minimizes reconstruction error, and hence maximizes variance
  • Extension of this derivation to handle multiple directions
  • Robust versions of PCA (eg: L1 PCA)
  • Applications of PCA in image compression
  • Slides (check moodle)
8th March
  • Relationship between PCA of tiny image patches and the DCT
  • Concept of overcomplete dictionaries: sinusoids and spikes examples
  • Issues of uniqueness and sparsity of representation in overcomplete dictionaries, dictionary learning as a generalization of K-means
  • A basic algorithm for simultaneously obtaining dictionary and sparse code - using projected gradient descent with adaptive step-size
  • Slides (check moodle)
10th March (extra lecture)
  • A basic algorithm for simultaneously obtaining dictionary and sparse code - using projected gradient descent with adaptive step-size
  • Method of Optimal Directions (MOD)
  • Union of Orthonormal Bases
  • KSVD method
  • Slides (check moodle)
12th March
  • KSVD method for compression, denoising, inpainting
  • Slides (check moodle)
14th March (extra lecture)
  • Concept of blind compressed sensing: the compressive KSVD method
  • Requirements for blind compressed sensing
  • Non-negative matrix factorization (NMF)
  • Non-negative sparse coding (NNSC)
  • Slides (check moodle)
2nd April
  • NNSC for Poisson noise; The Poisson noise model in imaging
  • Fisher's linear discriminant: case of 2 classes
  • Fisher's linear discriminant: case of more than 2 classes
  • Limitations of FLD, introduction to mutual information for classification
  • Slides (check moodle)
5th April
  • Limitations of FLD, introduction to mutual information for classification, concept of mutual information and its properties
  • Mutual information for optimal transform learning for classification
  • Kernel density estimation
  • Optimization of quadratic mutual information
Statistics of Natural Images
  • Power law
  • Correlation of pixel values in natural images
  • Statistics of DCT coefficients of natural images: laplacian models and justification for the same
  • Slides (check moodle)
7th April (extra lecture)
  • Statistics of natural image categories: natural, man-made,etc. Image and scene scale
  • A semi-automated method for reflection removal using statistics of image gradients: Iteratively reweighted least squares algorithm
  • Slides (check moodle)
9th April
  • A semi-automated method for reflection removal using statistics of image gradients: Iteratively reweighted least squares algorithm
  • Results of the IRLS algorithm for reflection removal, mixture of Laplacian prior for image gradients
  • Revision of basic Bayesian statistics, concept of MAP and MMSE, posterior probabilities with gaussian likelihood and Lapalcian and Gaussian priors
  • Compressed sensing based on statistical priors
  • Statistics of wavelet coefficients of natural images, use of dependencies between wavelet coefficients as priors for image denoising and in compression
  • Slides (check moodle)
12th April
  • Statistics of wavelet coefficients of natural images, use of dependencies between wavelet coefficients as priors for image denoising and in compression: linear model for variance of a wavelet coefficient given a set of neighbors
Low rank matrix recovery
  • Ubiquitousness of low rank matrices in image processing and machine learning: recommender systems, image patches assembled to form a matrix, distance matrices, applications in structrue from motion, applications in face recognition
  • The problem of low rank matrix completion
  • Slides (check moodle)
14th April (extra lecture)
  • The problem of low rank matrix completion
  • Informal statement of the basic theorem, concept of coherence of subspaces, sufficient conditions for successful recovery and pathological cases
  • Formal statement of theorem of low rank matrix completion
  • Empirical results for low rank matrix completion
  • Concept of low rank matrix recovery
  • Introduction to Robust Principal Components Analysis (PCA)
  • Motivating examples for RPCA: background subtraction in videos, specularity and shadow removal from face images
  • Slides (check moodle)
16th April
  • Concept of low rank matrix recovery
  • Basic theorems for low rank matrix recovery, concept of RIP of linear maps/operators, relation between low rank matrix recovery and low rank matrix completion
  • Introduction to Robust Principal Components Analysis (PCA)
  • Motivating examples for RPCA: background subtraction in videos, specularity and shadow removal from face images
  • Basic theorem for RPCA
  • Basic theorem for RPCA with incompletely observed entries
  • Slides (check moodle)
19th April
  • Singular Value Thresholding Algorithm (SVT) for matrix completion
  • Augmented Lagrangian Method (ALM): ALM for RPCA
  • Compressive RPCA: Greedy Algorithm by Waters et al, applications in video and hyperspectral CS, and RPCA with missing entries
  • Basic Theorem for Compressive RPCA
  • Slides (check moodle)