Eurographics 2010 Computational photography
Reinterpretable Imager: Towards Variable Post-Capture Space, Angle and Time Resolution in Photography
The idea
Capture once.
Decide later.
Take a photo, then choose how to interpret it in software: a detailed still image, a short video, or a refocusable light field. A dynamic aperture mask and a static mask near the sensor let spatial, angular, and temporal resolution be traded after capture, with different choices for different parts of the scene.
- Space
- Recover a high spatial resolution image
- Angle
- Refocus static scene content after capture
- Time
- Recover motion as a sequence of video frames
- One exposure
- Choose the interpretation for each part of the scene
01 / Results
Motion and refocusing in the same photo
A doll rotates on the right while the other dolls and the flowerpot remain still. From one exposure, we recover a video of the moving doll and light field information for the static scene.
Optical design
A coded aperture camera places a static mask in the aperture for applications such as extended depth of field. A high-frequency mask close to the sensor can capture light fields through optical heterodyning. Both designs were explored in our SIGGRAPH 2007 work. The Reinterpretable Imager combines a dynamic aperture mask with a static near-sensor mask, moving a pinhole or slit across the aperture during the exposure.
Moving a pinhole maps temporal variations in the scene to angular variations in the aperture. The near-sensor mask captures these as a light field. For a dynamic scene in focus, the sub-aperture views represent successive moments in time. For static scenes, those views represent angular variation and enable digital refocusing. The same captured photograph can therefore be reinterpreted as a short video or a light field.
Moving a vertical slit maps temporal variation to horizontal angular variation, while scene angle is encoded in the vertical direction. The resulting views contain time in one dimension and angle in the other, allowing digital refocusing in one dimension on an object moving in depth.
Implementation
The prototype uses a medium-format Mamiya camera with a high-frequency mask over the CCD to capture light fields through optical heterodyning. The near-sensor arrangement follows our SIGGRAPH 2008 work on glare reduction.
A motor-driven plastic wheel carries the printed aperture patterns. During a single exposure, the wheel rotates so that light is gathered through each pinhole or slit for an equal amount of time.
Abstract
We describe a novel multiplexing approach to achieve tradeoffs in space, angle and time resolution in photography. We explore the problem of mapping useful subsets of time-varying 4D light fields in a single snapshot. Our design is based on using a dynamic mask in the aperture and a static mask close to the sensor. The key idea is to exploit scene-specific redundancy along spatial, angular and temporal dimensions and to provide a programmable or variable resolution tradeoff among these dimensions. This allows a user to reinterpret the single captured photo as either a high spatial resolution image, a refocusable image stack or a video for different parts of the scene in post-processing.
A light field camera or a video camera forces an a priori choice in space-angle-time resolution. We demonstrate a single prototype which provides flexible post-capture abilities not possible using either a single-shot light field camera or a multi-frame video camera. We show several novel results including digital refocusing on objects moving in depth and capturing multiple facial expressions in a single photo.