open simultaneous real-time visible and infrared real-time visible and infrared video with...

Download OPEN Simultaneous real-time visible and infrared real-time visible and infrared video with single-pixel

Post on 10-Jun-2018




0 download

Embed Size (px)


  • 1Scientific RepoRts | 5:10669 | DOi: 10.1038/srep10669

    Simultaneous real-time visible and

    infrared video with single-pixel

    detectorsMatthew. P. Edgar1, Graham M. Gibson1, Richard W. Bowman2, Baoqing Sun1,

    Neal Radwell1, Kevin J. Mitchell1, Stephen S. Welsh1 & Miles J. Padgett1

    Conventional cameras rely upon a pixelated sensor to provide spatial resolution. An alternative

    approach replaces the sensor with a pixelated transmission mask encoded with a series of binary

    patterns. Combining knowledge of the series of patterns and the associated filtered intensities, measured by single-pixel detectors, allows an image to be deduced through data inversion. In this

    work we extend the concept of a single-pixel camera to provide continuous real-time video at 10 Hz

    , simultaneously in the visible and short-wave infrared, using an efficient computer algorithm. We demonstrate our camera for imaging through smoke, through a tinted screen, whilst performing

    compressive sampling and recovering high-resolution detail by arbitrarily controlling the pixel-

    binning of the masks. We anticipate real-time single-pixel video cameras to have considerable

    importance where pixelated sensors are limited, allowing for low-cost, non-visible imaging systems

    in applications such as night-vision, gas sensing and medical diagnostics.

    When multi-pixel sensors are not available due to cost or technological constraints, imaging systems require alternative approaches. A number of related approaches use spatially structured illumination13 or structured detection4,5 and a single-pixel detector to deduce an image. Perhaps the most obvious of these approaches is to raster scan a spatially selective detector over the field of view and rely upon the temporal analysis of the back-scattered light to give the intensity of every pixel in the image. Although giving a complete image, clearly this approach has an optical efficiency that scales inversely with the number of pixels in the image.

    Instead of scanning a single detector over the whole image an alternative is to measure many pixels simultaneously, which in practice improves the measurement signal-to-noise. Broadly-termed aperture coding, one implementation scheme uses a series of binary transmission masks applied using a spatial light modulator to the image formed by a lens, and a single detector to measure the transmitted inten-sity. The known series of mask patterns and the measured intensities can be combined and inverted, using a variety of algorithms, to give a good estimate of the image. The use of a single detector to obtain the image data leads this technique to be called a single-pixel camera4,6. We note at this point that the single-pixel camera has much in common with the field of computational ghost imaging711, whereupon the latter uses knowledge of projected illumination patterns and measured back-scattered signals. Both single-pixel cameras and computational ghost imaging systems use similar algorithms applied to the back scattered intensities to deduce the image.

    For both aforementioned approaches utilising single-pixel detectors to reconstruct images there exists a trade-off between acquisition time and image resolution, which results from the finite display rate of the spatial light modulator. Importantly however, most natural images exhibit similar characteristics, for example sparsity in their spatial frequencies, which allow for compressive techniques to represent images with less information. Indeed the field of compressed sensing asserts that an image can be recovered with far fewer measurements than the Nyquist limit. Recently, there has been considerable interest in the

    1SUPA, School of Physics and Astronomy, University of Glasgow, Glasgow, G12 8QQ, UK. 2Department of Physics, Cavendish Laboratory, University of Cambridge, Cambridge CB3 0HE, UK. Correspondence and requests for materials should be addressed to M.P.E. (email:

    received: 30 December 2014

    Accepted: 22 April 2015

    Published: 22 May 2015



    2Scientific RepoRts | 5:10669 | DOi: 10.1038/srep10669

    development of advanced compressive algorithms for the acquisition of video1220. However impressive the compression rates of these approaches, the associated reconstruction times greatly exceed the acqui-sition time thereby prohibiting use in real-time video systems for human interfacing.

    ResultsIn this work we demonstrate real-time video from a single-pixel camera for visible and short-wave infra-red (SWIR) wavelengths without the requirement for lengthy post-processing. Our system utilises a high-speed digital micro-mirror device (DMD) to apply spatial masks from which the transmitted light is spectrally filtered simultaneously onto four separate single-pixel detectors, corresponding to the red, green, blue and SWIR colour channels. We note that this experimental system has a similar configuration to that in Ref [4,6], with the addition of simultaneous spectral filtering. However, key to this approach is the implementation of an efficient iterative computer algorithm that stores only the current frame in memory and thus can operate continuously, which is an important feature for this technology in appli-cations. We compare real-time colour video (red, green, blue) and SWIR video (800-1800 nm) acquired and processed at frame rates of ~10 Hz with a resolution of 32 32 pixels or ~2.5 Hz at a resolution of 64 64 pixels. Furthermore, we demonstrate the use of real-time image optimisation similar to tech-niques used in existing compressed sensing work4,21,22, in the presence of moderate and excessive noise. In addition, we demonstrate the recovery of high-resolution detail in real-time by arbitrarily modifying both the DMD region of interest and mirror binning.

    Our modified single-pixel camera demonstrates simultaneous acquisition of multispectral images in the visible and shortwave infrared. Obtaining colour and SWIR images provide an intuitive demonstra-tion of the technology, showcasing the perfect pixel registration inherent with this approach. However, in principle these types of systems can be extended for imaging at mid-infrared and terahertz wavelengths, where existing detector arrays are very expensive or have inherent limitations, such as microbolome-ter arrays which require cooling for improved sensitivity or low-resolution Schottky diode arrays that require scanning.

    The optical design concept is based upon a high-speed digital light projector for which the light source is replaced with a detection system incorporating a hot mirror to separate SWIR and visible light. Subsequently, a dichroic prism is used to separate visible light into red, green and blue spectral bands. Photomultipliers (PMTs) are used to sense the visible colour channels and an InGaAs photodiode is used to sense the SWIR light (see Fig. 1). The controller board for the DMD allows binary patterns to be preloaded and then displayed at a maximum rate of 20.7 kHz. Each mirror on the DMD can be electronically actuated to one of two states representing on or off (transmissive or opaque)23. For every pattern displayed, the controller board also provides an output synchronisation TTL signal that is used to trigger signal acquisition on a high-dynamic range, analogue-to-digital converter (ADC), capable of acquiring 250 k samples/s.

    Figure 1. Experimental setup for real-time video with single-pixel detectors. A lens is used to form an image of the scene onto the digital-micromirror-device (DMD). The spatially filtered light is directed onto a hot-mirror, which reflects SWIR light onto an InGaAs photodetector (PD) and transmits visible light. A dichroic prism (x-prism) subsequently filters the visible light into red, green and blue output ports, sensed by three identical photomultipliers (PMs). The measured intensities are digitised by an analog-to-digital converter (ADC) for computer processing.


    3Scientific RepoRts | 5:10669 | DOi: 10.1038/srep10669

    As with any imaging system, the image quality is dependent upon the level of noise on the detector, whether it be a single-pixel or a sensor array. With single-pixel imaging technology, image quality can be optimised via both optical and computational techniques. Similar to the work of Ref [4,6], we perform optical optimisation by making use of PMTs as the single-pixel detectors for detection of visible light, which enables the system to operate under low-light conditions. In this experiment we make use of the well-known Hadamard matrices24, which are binary matrices that form a complete orthonormal set and where each pattern contains an equal number of 1s and 1s, representing on and off respectively for each mask applied to the DMD. Each row from the Hadamard matrix, when reshaped to 2D, contains a different subset of spatial frequencies, such that an N N pixel image can be fully sampled with N Hadamard patterns. However, to preserve orthogonality, this approach relies on sensing the light reflected in both outputs of the DMD, or alternatively, by displaying each pattern and immediately succeeding it by its inverse. The former requires the use of two detectors for each waveband, while the latter approach used here halves the maximum achievable frame rate. Both approaches utilise the difference between the two measurements to provide a zero-mean differential signal25, which is similar to lock-in detection at 22 kHz and 11 kHz respectively, helping to eliminate unwanted sources of noise.

    Given a sequence of N orthonormal pattern pairs Ai,j, where the pixel values are assigned 1 (i is the pixel number a


View more >