Summary
Take a photo: Decide later if it stays a photo, becomes a video or turns into a lightfield. Ultimate post-capture control in software.
Abstract
We describe a novelmultiplexing approach to achieve
tradeoffs in space, angle and time resolution in photography. We
explore the problem of mapping useful subsets of time-varying 4D
lightfields in a single snapshot. Our design is based on using a
dynamic mask in the aperture and a static mask close to the sensor. The
key idea is to exploit scene-specific redundancy along spatial, angular
and temporal dimensions and to provide a programmable or variable
resolution tradeoff among these dimensions. This allows a user to
reinterpret the single captured photo as either a high spatial
resolution image, a refocusable image stack or a video for different
parts of the scene in post-processing.
A lightfield camera or a video camera forces a-priori choice in
space-angle-time resolution. We demonstrate a single prototype which
provides flexible post-capture abilities not possible using either a
single-shot lightfield camera or a multi-frame video camera. We show
several novel results including digital refocusing on objects moving in
depth and capturing multiple facial expressions in a single photo.
Paper (Preprint)
High res pdf (62
MB), Low res pdf (5MB)
Video, avi, 50
MB
Talk Slides
Results
1. Capturing a video in a single shot. In this example, we trade-off spatial resolution to capture temporal resolution in a single photo.
(Left) A single captured photo using our camera, where a
person is
making facial expressions within the exposure time. (Right) Low spatial
resolution video frames recovered from the photo.
2. This example shows how we can have different resolution for
different parts of the image, not possible with a conventional camera.
In this example, there are several static dolls and static flowerpot in
the back. During the exposure of the photo, a doll is rotated on the
right. From the single captured image, we can obtain a video for
the
rotating doll and lightfield information for the static parts of the
scene.
 Single Captured Photo |
 GIF animation shwoing obtained video (9 frames for rotating doll)  GIF animation shwoing digital refocusing on static dolls and flowerpot. The focus is changing from back to front in 4 steps. |
Optical Design
The optical design of Reinterpretable Imager is
simple
to
understand. In a Coded Aperture camera, a static mask is placed in the
aperture for applications such as extended depth of field. This design
was showed by our group in SIGGRAPH 2007.
To
capture
lightfields,
one
can
insert
a
high
frequency
mask
close
to
the
sensor
to
achieve
optical
heterodyning. This design was also showed
by our group in SIGGRAPH 2007. Reinterpretable imager uses a dynamic mask in the aperture and a
static near-sensor mask. For dynamic aperture mask, we either move a
pinhole or a slit across the aperture during the exposure time of the
camera to capture the photo.
By moving the pinhole in the aperture during the expsoure time of the
camera, the temporal ray variations in the scene are mapped to angular
variations in the aperture. These are captured by the mask close to the
sensor as a light field. Thus, if the scene only consist of temporal
variations (a dynamic scene in focus), temporal frames can be captured
as sub-aperture "views" of the light field. For static scenes, the
sub-aperture "views" of the light field correspond to the angular
variations, using which digital refocusing can be done. Thus, depending
on the scene, the captured photo can be "reinterpreted" as a short
video or a light field.
Similarly, by moving a vertical slit in the apertur, the temporal
variations in the scene are mapped to horizontal angular variations in
the aperture. And angular variations in the scene are mapped to
vertical angular variations in the aperture. Thus, the captured light
field views correpsond to temporal variations in one dimension and
angular variations in other dimension. This can be used to digitally
refocus (in 1D) on an object moving in depth.
Implementation
Our implementation
is as shown above. We use a
medium
format Mamiya camera as the sensor. We insert a high frequency mask on
top of the CCD for capturing lightfields via optical heterodyning. This
arrangement
is similar to as shown previously in our SIGGRAPH 2008 paper on glare reduction. The more interesting
part is the implementation of the dynamic mask in the aperture. We use
a motor drive plastic wheel, on which printed plastic pattern as shown
above are attached. Note that we capture only a single photo. During
the exposure time of the camera, the motor rotates the wheel so that
light is gathered through each pinhole (or slit) for equal amount of
time. This video shows the motion of the wheel during the exposure time.