CS784 - Image Synthesis

Semester 2 (Spring), 2023-2024


Overview

This course deals with the process of image formation or synthesis on a computer. We will begin with detailed discussions of the classical methods from Computer Graphics for rendering and cover this exciting and fast changing field through its modern inception in the era of neural rendering and generative models.

This is NOT a course in deep learning or machine learning. This is a course in visual computing. Students are expected to have done a first course in either of computer graphics, image processing or computer vision.

The course is meant to be a very hands-on course and will include a healthy mix of theory, programming assignments and assignments that require working in open source digital content creation (DCC) platforms like Blender, among others.

Teaser Video

Logistics


Registration Instructions

  • This is open to students from all departments who have a prior background in either of computer graphics, image processing, or computer vision. Prior background in introductory machine learning is also assumed. Note that none of these are stirctly enforced, but are strongly desired. If you take the course without requisite background, you do so at your own risk.

  • You are not allowed to register for this courses if you have done another course that overlaps significantly in content with it. If you have a doubt about this, ask me.

  • Any B.Tech., DD or IDDDP student who is in the 3rd year or above can take the course. Please do not take the course if you are an undergraduate student in the second year or earlier. Please make sure you have the requisite background as given above.

  • Any M.Tech., M.S. or Ph.D. student can take the course. Please make sure you have the requisite background as given above.

  • You can tag the course as whatever you want.
  • Audits are not allowed.

Eligibility/Prerequisites

If you are curious about image synthesis at all - attend the first lecture.

  • Prior knowledge of the following is strongly recommended

    • C/C++ and Python Programming.
    • Basic Linear Algebra is desirable.
    • Data structures and algorithms.
    • Computer Graphics, Image Processing or Computer Vision.
    • Introductory Machine Learning.

  • Learn how to use git.

  • Make sure you have a GitHub id. You will need it for the course.


Course Content

Here is a preliminary list of topics for the course. The contents may change as the course progresses.

  1. Global Illumination and Rendering - Radiometry, Photometry, Ray-tracing, Surface reflectance and BRDFs, Monte Carlo Path Tracing - Importance Sampling, Metropolis Light Transport, Photon Mapping, Ambient Illumination, Shaders, Volume Rendering

  2. Image-based Rendering - Planar panoramas, View Morphing, Novel View Synthesis, Plenoptic function, Lightfields,

  3. Neural and differentiable Rendering - Neural scene representation and rendering, Neural Radiance fields, Differentiable rendering, Inverse rendering, Image synthesis from text, audio and other images, generative models for images

  4. Image Synthesis for Virtual and Augmented Reality - Spherical (360) Panoramas, Neural Antialiasing, Latency, Vergence-Accommodation Conflict and other causes of discomfort in VR, Foveated rendering

  5. Real or synthetic - Statistics of real images, Methods for spotting fake images, Responsible use of synthesized images


Lecture Schedule

Lecures slides for the course can be found on IIT Bombay Moodle.

Additional reading material for all topics can be found here.

Date Topics Tasks and Resources
Jan 5 Introduction to the Course
Jan 9
Jan 12		 Digital Light - Pixels, Light and Colour For additional readings, see here.
Jan 15
Jan 19
Jan 23		 Ray Tracing Assignment 1 announced on Jan 23.
Jan 30 Radiometry Assignment 1, Part 1 due today.
Feb 2 BRDF For additional readings, see here.
Feb 6
Feb 9		 Monte Carlo Path Tracing For additional readings, see here. Read sections from the PBR book on estimator efficiency increasing methods and sampling of multidimensinal distributions.
Feb 13 Ambient Occlusion, Photon Mapping and Point-based GI Assignment 1, Part 2 deadline is over. Assignment 2 has been announced.
Feb 16 Volume Graphics
Feb 20 Doubts Clearing Class
Feb 28 Mid Semester Exam CC 105, 2:30pm to 4:30pm
Mar 5 Image Based Rendering
Mar 12 Lightfields Read Understanding other cameras using Lightfields. See here.
Mar 15 Neural Radiance Fields Read the original paper by Mildenhall et al. and Volume Rendering for NeRFS. See here.
Mar 19 Instant NGP Read the original paper by Muller et al. See here.
Mar 23
Mar 26		 Augmented and Virtual Reality
Mar 29
Apr 2		 Image Synthesis for MR For additional readings, see here.
Apr 5 3D Gaussians for Rendering Radiance Fields Read the original paper by Kerbl et al. See here.

Assignments and Homeworks

Assignments for the course can be found on IIT Bombay Moodle

Resources

  1. Recorded Videos on YouTube for Computer Graphics (CS475/CS675)

    1. Recorded videos for the basic course on Computer Graphics can be found on YouTube here.

  2. Books

    1. Physically Based Rendering (4ed, 2023) 302226 from theory toi Implementation, Matt Pharr, Wenzel Jacom and GregHumphreys, MIT Press.
    2. Principles of Digital Image Synthesis (1995), Vol 1 & 2,Andrew Glassner, Morgan Kaufmann.
    3. Computer Vision: Algorithms and Applications (2ed, 2023), Richard Szeliski, Springer.
    4. Understanding Deep Learning (2023), Simon J. D. Prince, MIT Press.

  3. Online Readings

    1. Neural Rendering
      a. Course on Neural Rendering from CVPR 2020

  4. Resources

    1. Pixels, Light and Colour

      1. A Biography of the PIXEL, Alvy Ray Smith
      2. Introduction to Light, Color and Color Space
      3. CIE XYZ and xyY Color Spaces Douglas A. Kerr
      4. Perceptually Uniform Color Models, interactive visualization, Max Bo
      5. Causes of Color

    2. Ray Tracing

      1. Ray Tracing
      2. The Internet Ray Tracing Competition (the competition seems to be down at the moment)
      3. The Persistence of Vision Raytracer
      4. Object-Object Intersections
      5. Overview of raytracing on Scratchapixel

    3. Radiometry and BRDF

      1. Fresnel Reflectance
      2. Data from Cornell Light Measurement Lab
      3. Experimental Validation of Analytical BRDF Models, Ngan, Durand, Matusik
      4. Notes on Radiometry by Prof. Steve Marschner, Cornell University

    4. Radiosity

      1. SIGGRAPH Education Radiosity Slide Set
      2. Modeling the interaction of light between diffuse surfaces, Goral, Torrance, Greenberg and Battaile, SIGGRAPH 84.
      3. The hemi-cube: a radiosity solution for complex environments, Cohen and Greenberg, SIGGRAPH 85.
      4. RADIOSITY: An Illuminating Perspective, Drucker, Written Doctoral Exam

    5. Monte-Carlo Raytracing

      1. smallpt: G.I. in 99 lines of C++
      2. The Global Illumination Compendium
      3. State of the Art in Monte Carlo Ray Tracing for Realistic Image Synthesis, SIGGRAPH 2001 course notes

    6. Ambient Occlusion, Photon Mapping, Point-based GI

      1. Point-based Global Illumination, Per H. Christensen, PIXAR
      2. A practical guide to using Photon Maps, Henrik Wann Jensen, SIGGRAPH 2000 Course
      3. Ambient Occlusion, Wikipedia
      4. Ambient Occlusion, GPU Gems, CHapter 17, NVIDIA

    7. LightFields, Neural Rendering

      1. Light field rendering, Marc Levoy and Pat Hanrahan, SIGGRAPH 1996.
      2. The Lumigraph, Gortler et al. SIGGRAPH 1996.
      3. Dynamic Reparameterized Lightfields, Isaksen et al., SIGGRAPH 2000.
      4. Understanding other cameras using Lightfields (discussed in class), Levin et al., ECCV 2008
      5. Refocusing and Lightfields, Fredo Durand and Bill Freeman, Slides from course on computational photography
      6. Light Field Imaging: The Future of VR-AR-MR-Part 1, Paul Debevec
      7. Why is Neural Rendering so Cool?, Talk by Matthias Neißner, TUM.
      8. Neural Radiance Fields Mildenhall et al., ECCV 2020
      9. Volume Rendering for NeRFS, Tagliasacchi and Mildenhall, 2022
      10. Understanding and Extending Neural Radiance Fields, Talk by Jonathan T. Barron, Google Research.
      11. Neural Fields in Computer Vision, CVPR 2022 Tutorial.
      12. Instant NGP, Muller et al., SIGGRAPH 2022.

    8. Augmented Reality

      1. Website of the Augmented Reality Book
      2. Google Spotlight Stories
      3. Spherical Harmonic Lighting - The Gritty Details, Robin Green, 2003

    9. 3D Gaussians for Radiance Field Reconstruction

      1. Surface Splatting, Zwicker et al., SIGGRAPH 2001
      2. EWA Volume Splatting, Zwicker et al., IEEE VIS 2001
      3. 3D Gaussians for Real-Time Radiance Field Rendering, Kerbl at al., SIGGRAPH 2023

Last updated on Dec 17, 2023