Project Title: Image Relighting using Polynomial Texture Maps
Team:
  Ameya Ranade (05005014)
  Sree Shankar (08305023)
  Sriram Kashyap (08305028)

Polynomial Texture Maps (PTMs) are textures where coefficients of a biquadratic polynomial are stored per texel, and used to reconstruct the surface color under varying lighting conditions. Like bump mapping, this allows the perception of surface deformations. However, the PTM technique is image based, and photographs of a surface under varying lighting conditions can be used to construct these maps. Unlike bump maps, these PTMs also capture variations due to surface self­shadowing and interreflections, which enhance realism.

Typically in a PTM there are nine values stored per texel. The first three are the red, green, and blue color values. The next six values are coefficients to a biquadratic equation. This equation takes a given light position in relation to the texture and calculates the luminance for that texel. This function is a simplification of the BRDF for that texel. Finally, the color of the texel is modulated by the luminance.

Faces rendered using our relighting application

Click here to Download the project files

How to run the application

• In the command prompt, go to the program's directory (this directory contains the data,ptm,Jama directories)
• Type 'java ptm.Main'
• Wait for about 5-10 seconds for the application to load (delay due to time taken to compute the PTM)
• The light source can be moved by dragging the mouse over the window that appears (if the window becomes white in some places, please click and drag the mouse over the window to fix it)

Source Files Description

• The ptm directory contains the class files developed and compiled by us
• The Jama directory contains the class files of a Java Matrix Library
• The data directory contains data sets (one in each sub-directory)
• The init_ptm.txt file is a config file
• The file 'source.tar.gz' contains the source code needed to compile the program

Input Data Description

• Image Data: a set of sequentially numbered image files
• Light Data: a file containing r-theta-phi values of where the light source was in each image. The first line is the number of images in data set. The second line is the format of the remaining data. This is either 'xyz'(x,y,z coordinates) or 'rtp'(r,theta,phi coordinates)
• Render Data: a file containing the re-lighting information. First line is a count of number of lights used for re-lighting. Each line is a light source. The first light source can be controlled with mouse. The remaining are static (we have not yet implemented a system to move them all, but their light contribution is calculated real-time). The values in each line are : r,theta,phi,red,green,blue,intensity, in that order

• Load N input images and light positions
• Compute an initial matrix A(Nx6) of this form:
• • Each row represents an input image
  • The row contains the values: 1, theta, theta*theta, theta*phi, phi, phi*phi
For each pixel, compute a matrix I(Nx1) of this form:
• Each row contains the intensity of the pixel as seen in that image
• For each pixel, compute the coefficient matrix C(6x1):
• The rows of the matrix are coefficients of the polynomials represented by A
• These 6 coefficients are stored for each pixel
• Computation is by calculation of pseudo inverse:C=(inverse(transpose(A)*A))*transpose(A)*I
• Finally, calculate the base RGB color of the pixel using a weighted average. The weight is highest for medium intensity pixels(intensity is approximately 0.5), and lowest for max and min intensities (1 and 0). This way, we get a good approximation of the color of the entire object, even though the entire object may not be clearly visible in any one image
• To find the intensity at each point, we evaluate the above polynomial. This intensity is used to calculate the final color of the pixel
• To implement colored lights, we assign an r-g-b value to each light source, and multiply the pixel's component values by the light's component values, at each pixel

During the course of our project, we had come across many obstacles, some expected and and others unexpected. Given below are some of the major ones we faced and what we did to overcome those.

• Computing the pseudo inverse of the matrix. In several cases, the matrix was singular when we considered theta and phi values directly. This problem was not faced in the paper [1] because the input data set they took was from highly specific locations. To overcome this, we took two measures:
• Project the theta-phi ray onto the texture plane to get a x,y coordinate
• Add some random noise to the input parameters (upto + or - 5 degrees)
• Calculating the base color of the image. The entire image color has to be constructed from many pieces taken from different images
• We developed a weighted averaging method, where higher weightage is given to pixels with moderate intensity, an lower weighage is given to bright/dark pixels
• Java Performance issues: Default Java pixel manipulation is extremely slow, even with the BufferedImage object
  • We developed a FastImage class that allows direct access to the image color buffers
• The real challenging part was in getting the real life images. We have tried several techniques to capture real data sets. Some of them are:
• Using a box with holes in it to let light through: The problem was that the boxes were too small and we could not have the light source far enough from the object (the object would get partially lit)
• Placing the object in front of wall,and moving the torch, keeping the camera constant: The problem here was that there was not enough light coverage, and the shadows cast on the wall caused artifacts in the output
• Using the above 2 methods were not giving us satisfactory results. So then we placed the object on a tall thin object in the middle of a dark room. This way we were getting better results as compared to the above 2 methods
• We still were getting issues regarding the light source (We used a white light torch) as in the places were the light fell on the image was becoming too bright. So we removed the convex lens (which causes the concentrating of the light from the torch), after which we finally started getting better input images
• We also faced issues of the camera not being rigidly still while taking the images. For best results, it was of utmost importance that the object remained in exactly in the same plane in all the images. even a shift of 4 to 5 pixels could play havoc with the end result. We overcame this problem by taking the video of the entire process of taking input images at different angles and then taking the required snapshots

[1] Polynomial Texture Mapping. T Malzbender, D Gelb, H Wolters, Proceedings of ACM Siggraph,2001