Removing Photography Artifacts using Gradient Projection and Flash-Exposure Sampling: SIGGRAPH 2005

We propose a strikingly simple technique to remove artifacts such as glass reflections from images using a gradient domain method based on gradient projection. The basic idea is that the direction of the image gradient can be considered as a quasi-invariant with respect to illumination changes. While photographing a scene through a glass or reflective layer, the image often has annoying reflections of objects in front of the glass. We use a pair of flash/no-flash (ambient) image to remove these reflections.

Consider a pair of flash/no-flash images as shown on the right. The ambient image has reflections of the photographer who is taking this picture. You can see the hands and face of the photographer and even the tripod getting reflected in this image. The flash image is a short exposure image which removes these reflections but has a hotspot. This hotspot is caused by the direct reflection of flash from the glossy surface of the painting. We wish to combine these two images to get an image free of ambient reflections and flash hotspot. Consider a highlighted patch in these images. This patch does not have any reflections. So the direction of the ambient and flash image gradient vector will be the same for any pixel in this patch.
Now consider another image patch at the bottom which overlaps with reflections. The direction of the gradient vector in flash and ambient images will not be the same. Due to the reflections in the ambient image, the direction of the ambient image gradient vector is perturbed.
Since we know that the direction of the image gradient in flash and ambient images should remain the same, we can constrain the direction of the ambient image gradient vector by taking a vector projection of the ambient image gradient on to the flash image gradient vector. The "result" image is obtained by a 2D integration of the projected gradient field. Interestingly, the residual gradients removed by taking the projection can be integrated to obtain the "reflection layer". Thus by using gradient projection, we can decompose a image with reflections into a reflection free image and the reflection layer.
Why do Projection? One can ask why do we need to do a projection. We know the direction of the gradient vector so we can choose some other magnitude for it. The important thing is to maintain the same direction. The reason for doing projection is that for a given gradient field, the gradient field which lies orthogonal to it does not have much visual information. Suppose (gx,gy) denotes the original gradient field and (gx_90,gy_90) denotes the gradient field obtained by rotating each gradient vector by 90 degrees. One can see that the divergence of the rotated gradient field is zero everywhere. So the reconstruction from the rotated gradient field will be zero everywhere. We took the Lena image and find gradients in x and y directions. Then we start rotating these gradient vectors by certain angles and reconstructing from the rotated gradient field. One can see that at a rotation angle of 90 degress, the reconstruction does not have any visual information. At rotation angle of 180 degrees, the obtained image is the negative of the original image.
Also see My CVPR 2006 paper for a more general algorithm for edge suppression by gradient field transformation.