260 — Casual 3D Photography

Many systems exist to produce 3D scans of a scene, but these methods are often either expensive or they require uncommon hardware or software. For example, the Lytro cameras — the manufacturer of which was purchased and shut down, as far as I can tell, by Google — generate light-field photography, but they require a Lytro camera handheld, and are not compatible with conventional photography hardware or software.

This suite of technology instead uses structure-from-motion (SfM)-based algorithms to develop a two-layer model of a scene: The “Front” layer is a photographic representation of the pixels visible to the camera from a static, centralized panoramic viewpoint. The “Far” layer is generated by determining the 3D positions of pixels that are not visible from the initial perspective. It is this layer which gives significant depth to the scene, adding the extra information that is required to prevent empty space between regions occluded in the panorama’s original frame.

The two layers are connected by hallucinating the interpolated colors of the missing faces that connect the front and back layers; a graph stitching operation converts the sum of these layers into continuous regions where appropriate (and separates meshes where stitching is implausible).

The results from this work are stunning, and improve greatly upon the existing work I’ve seen (such as Google’s Lens-Blur system, which is also pretty impressive).