Building a Handheld Light Field Camera

A conventional camera sums the rays arriving at each sensor pixel. A light field camera also records how those rays vary with direction. A sharply focused scene point sends a cone of rays through the lens to one image location; sampling that cone reveals angular information.

Ray space has four dimensions: two sensor-plane coordinates (x, y) and two directional coordinates, such as azimuth and elevation. This description ignores attenuation along rays and wavelength and time dependence. A conventional 2D image integrates the 4D ray space over the angular dimensions.

A lenslet array can redirect rays onto separate sensor pixels. In the plenoptic design demonstrated by Ren Ng and colleagues in 2005, the main lens focuses on the lenslet array, which focuses onto the sensor. The original guide compared this approach, later commercialized by Lytro, with mask-based capture. The Stanford prototype used a medium-format Contax camera and produced roughly 300 × 300 spatial samples and 14 × 14 angular samples.

The mask-based design attenuates rays instead of refracting them. A pinhole array near the sensor forms an image of the aperture beneath each pinhole, sampling angular variation. This is part of the broader history of integral imaging, which began with Lippmann’s 1908 proposal.

The prototype used a Mamiya 645ZD with a 22-megapixel digital back: a 36 × 48 mm Dalsa CCD with 5344 × 4008 pixels. A 1.2 mm protective glass layer covered the sensor.

A matching pinhole-array transparency was placed on the protective glass. An additional glass piece flattened and held the mask in place. The original demonstration reported insertion in less than a minute.

The masks were printed at up to 5080 DPI with square pinhole openings 25 microns wide. At the time of the experiment, an A4 transparency carrying 20 masks cost less than $100, or about $5 per mask. These are historical prototype costs.

The original printing reference was Pageworks.

Masks are inexpensive to print, easy to replace, and do not require each lenslet to focus an aperture image onto the sensor. The mask must still be parallel to the sensor. Lenslet cameras require precise positioning to preserve angular resolution.

Single-shot light field capture trades spatial resolution for angular resolution: sensor pixels = spatial samples × angular samples. A 5 × 5 angular sampling grid reduces spatial resolution by a factor of five in each direction, whether masks or lenslets are used.

A planar Lambertian scene in sharp focus can be a special case. Since it carries no extra angular variation, a full-resolution 2D image can be recovered by dividing the masked photograph by a calibration photograph of a uniform light box. The mask blocks some rays but does not remove spatial information from the focused image; calibration compensates for pixel-dependent attenuation.

The principal disadvantage is lost light. The prototype pinhole array transmitted about 5% of the incoming light. The original outdoor experiments used exposures around 0.5 seconds, which can introduce motion blur; the demonstrated glare-reduction examples used static scenes. A sum-of-cosines mask can increase throughput.

The SIGGRAPH 2008 experiments used uniform and randomized pinhole arrays. A uniform array supports straightforward light field recovery. Randomizing pinhole positions disrupts the spatial structure needed for that reconstruction, but still supports statistical glare reduction. The randomized design can therefore reduce glare without reconstructing a full light field or accepting its usual spatial-resolution loss. See Figure 8 of the paper.

Lippmann proposed integral imaging in 1908; early demonstrations followed in 1911, including Sokolov’s pinhole-aperture sheet. These early arrangements did not use a main lens to focus the scene onto the lenslets. Ives added a larger-aperture objective lens in 1930, and Kanolt explored pinhole arrays with a large objective in 1933. Coffey’s work described f-number matching between the main lens and lenslet design.

S. P. Ivanov and L. V. Akimakina performed lens-array experiments in 1948, followed by further lens-array designs. Ren Ng’s 2005 camera brought handheld plenoptic capture to consumer-photography experiments. Our group presented mask-based capture in 2007 and ray-sampling methods for glare reduction in 2008.

The historical papers and patents from the original archive are linked below.