Home

CVPR 2010 Computational imaging

Optimal Coded Sampling for Temporal Super-Resolution

Amit Agrawal, Mohit Gupta, Ashok Veeraraghavan, and Srinivasa Narasimhan

01

The idea

Low-speed cameras.
High-speed video.

Coded sampling combines multiple low-cost, low-speed cameras into an effective high-speed camera. Each camera uses a different temporal exposure code, gathering more light than short-exposure sampling while allowing efficient reconstruction of sharp video frames.

N× temporal resolution
N cameras, each running at frame rate f, recover video at Nf
N/2× more light
Compared with an equivalent high-speed camera
Small reconstruction system
An N × N linear system for each set of N output frames
Streaming reconstruction
Independent frame groups reduce computation and memory

02 / Sampling strategies

Three ways to capture fast motion

Given N cameras running at frame rate f, how can we recover video at frame rate Nf? The exposure pattern determines the balance between light collection, motion blur, and reconstruction quality.

Point sampling

Staggered short exposures across multiple cameras for point sampling
Short, staggered exposures of duration 1/(Nf).

Interleave sharp frames from the cameras to obtain N× the frame rate without post-processing.

Tradeoff: no light advantage over an equivalent high-speed camera.

Box sampling

Staggered long continuous exposures across multiple cameras for box sampling
Long, staggered exposures of duration 1/f.

Collect N× more light than an equivalent high-speed camera and remove temporal aliasing, but every captured frame has motion blur.

Tradeoff: suppressed high frequencies make reconstruction ill-posed. A large linear system amplifies noise, making the full N× frame-rate gain difficult to recover.

Coded sampling Our method

Different temporally modulated exposure codes across cameras for coded sampling
Long exposures with a distinct temporal code per camera.

Use a different code for each camera, repeated across its frames. Recover N× the frame rate with N/2× more light than an equivalent high-speed camera.

Advantage: codes maximize reconstruction SNR under additive Gaussian noise. A well-posed N × N system supports streaming reconstruction.

Tradeoff: N/2× the light of an equivalent high-speed camera, rather than the N× gain of box sampling.

03 / Results

Sharp frames from coded blur

Four cameras capture four input frames. Coded sampling reconstructs four output frames by solving a 4 × 4 linear system at each pixel.

Rotating fan experiment comparing captured and reconstructed frames for box sampling, regularized box sampling, and coded sampling
Box sampling versus coded sampling on a rotating fan. Box sampling is equivalent to an all-ones exposure code; coded sampling uses distinct exposure patterns to recover sharp frames with low noise.

Coded sampling allows streaming output with low computation. Box sampling requires a much larger linear system spanning several frames and increases reconstruction noise. Regularization reduces that noise at the cost of additional blur.

04

Abstract

Conventional low frame rate cameras result in blur and/or aliasing in images while capturing fast dynamic events. Multiple low speed cameras have been used previously with staggered sampling to increase the temporal resolution. However, previous approaches are inefficient: they either use small integration time for each camera which does not provide light benefit, or use large integration time in a way that requires solving a big ill-posed linear system.

We propose coded sampling that addresses these issues: using N cameras it allows N times temporal super-resolution while allowing N/2 times more light compared to an equivalent high speed camera. In addition, it results in a well-posed linear system which can be solved independently for each frame, avoiding reconstruction artifacts and significantly reducing the computational time and memory.

Our proposed sampling uses optimal multiplexing code considering additive Gaussian noise to achieve the maximum possible SNR in the recovered video. We show how to implement coded sampling on off-the-shelf machine vision cameras. We also propose a new class of invertible codes that allow continuous blur in captured frames, leading to an easier hardware implementation.

05