'Virtual camera' images around corners, bettering previous methods

06 Aug 2019

Group from Wisconsin-Madison and Zaragoza use new math approach with cameras to cut fuzziness, enable video.

Scientists have for some years been researching how to image objects through barriers or around corners with mixed results for a variety of approaches to “non-line-of-sight” imaging methods. Now a team of scientists from the University of Wisconsin–Madison, USA (UW), and Universidad de Zaragoza, Spain, claim this process is a step closer to reality.

They have drawn on the lessons of classical optics, and shown that it is possible to image complex hidden scenes using a projected “virtual camera” to see around barriers.

The technology, they describe as a “virtual camera”, is reported in Nature. The developers say that, once perfected, it could be used in a wide range of applications, from defense and disaster relief to manufacturing and medical imaging.

In medicine, for example, the technology holds promise for applications like robotic surgery, or where a surgeon’s field of view is restricted when doing sensitive procedures. The work has been funded largely by the military through the U.S. Defense Department’s Advanced Research Projects Agency and NASA, which sees the technology as a potential way to explore hidden features on the moon and Mars.

Clearer views, video

Non-line-of-sight imaging has typically been limited by technical challenges such as the fuzzy quality of derived pictures of relatively simple target scenes. Challenges that could be overcome by the new approach include imaging more complex hidden scenes, seeing around multiple corners and producing moving videos.

“This non-line-of sight imaging has been around for a while,” commented Andreas Velten, a professor of biostatistics and medical informatics in the UW School of Medicine & Public Health and the senior author of the Nature study. “There have been a lot of different approaches.”

The basic idea of non-line of-sight imaging, said Velten, revolves around the use of indirect, reflected light, a “light echo”, to capture images of the hidden scene. Photons from thousands of pulses of laser light are reflected off a wall or another surface to an obscured scene and the reflected, diffused light bounces back to sensors connected to a camera. The recaptured light particles or photons are then used to digitally reconstruct the hidden scene in three dimensions. “We send light pulses to a surface and see the light coming back, and from that we can see what’s in the hidden scene,” said Velten.

The work by the UW-Zaragoza group goes beyond simple scenes and addresses the primary limitations to existing non-line-of-sight imaging technology, including varying material qualities of the walls and surfaces of the hidden objects, large variations in brightness of different hidden objects, and the massive amounts of noisy data used to reconstruct larger scenes.

Advantageous algorithms

Velten and his colleagues, including Diego Gutierrez of Universidad de Zaragoza, decided to turn the problem around, by applying the same math used to interpret images taken with conventional imaging systems. Their new method surmounts the use of a single reconstruction algorithm and describes a new class of advantageous imaging algorithms.

Conventional systems, said Gutierrez, interpret diffracted light as waves, which can be shaped into images by applying established mathematical transformations to the light waves propagating through the imaging system.

In the case of non-line-of-sight imaging, the challenge of imaging a hidden scene, says Velten, is resolved by reformulating the non-line-of-sight imaging problem as a wave diffraction problem and then using established mathematical transforms from other imaging systems to interpret the waves and reconstruct an image of a hidden scene. By doing this, the new method turns any diffuse wall into a virtual camera.

“What we did was express the problem using waves,” said Velten, who is also affiliated with the Morgridge Institute for Research and the UW–Madison Laboratory for Optical and Computational Instrumentation. “The systems have the same underlying math, but we found that our reconstruction is surprisingly robust, even when working with bad data. You can do it with fewer photons.”

’Virtual camera’

The ability to project a camera from one surface to another suggests that the technology could be developed to a point where it is possible to see around multiple corners. “This should allow us to image around an arbitrary number of corners,” said Velten. “To do so, light has to undergo multiple reflections and the problem is how do you separate the light coming from different surfaces? This virtual camera can do that.”

The team says that the technology could be significantly improved if arrays of sensors can be devised to capture the light reflected from a hidden scene; the experiments so far, described in the Nature paper, depended on just a single detector.