# Single Pixel Camera - Compressive Sensing

I work with a setup that measures high frequency fluctuations in light using a photodiode. We steer the light over a sample as we measure these fluctuations.

I am familiar with compressive sensing where we only sample a few points to reconstruct the full signal. The sample is of course done in the same dimensions as the signal - for audio, the sampling is across time for instance. For an image, we randomly subsample in both x and y coordinates.

I came across a single pixel camera which employs a Digital Micromirror Device to project a pattern onto a scene which is then imaged with, again, a photodiode. Most of the descriptions I've read say that the each pattern would in effect sum the light from the scene and repeated scanning with random patterns can yield images with similar quality as raster scanning.

How are these images reconstructed? If we sum the light, we lose where the individual sums of that light came from. How does repeated acqusition solve this?

• The simplest method here is to use a Hadamard Transform, see Nat Methods . 2014 Nov;11(11):1131-4. doi: 10.1038/nmeth.3139. Epub 2014 Oct 5 for an example of how to generate and use these. An imaging example can be found in Harwit M, Sloane NJA. Hadamard Transform Optics. Academic Press; 1979. – porphyrin Jun 9 '20 at 9:05

First I explain how compressive sensing is leveraged into imaging reconstruction and then a little bit on how CS is deployed in an imaging hardware.

Compressive Sensing

For the sake of simplicity lets assume our image is 1D (i.e. a row of pixels). And assume the image you want to ultimately construct is $$X_N$$ with $$N$$ being length of it. The simple choice of course is to stack up $$N$$ sensors very close to each other and basically build up a 1D camera. However if you have only 1 image sensor, you need to either move the sensor in $$N$$ positions as if there are $$N$$ sensors or as you've done, steer light $$N$$ times and capture the pixel value. Now, the using compressive sensing you can do $$M$$ measurements ($$M << N$$) and still construct the image. Now lets assume each pixel from pixel number 1 to pixel number $$N$$ as unknowns of the following system of equations. The right side, $$Y$$ is simple an vector that each element of it is sum of a number of pixels 1 to $$N$$, e.g.

$$Y_1 = c_{1,1}\times P_1 + c_{1,2}\times P_2+..c_{1,i} \times P_i+...c_{1,N}\times P_N$$ $$Y_2 = c_{2,1}\times P_1 + c_{2,2}\times P_2+..c_{2,i} \times P_i+...c_{2,N}\times P_N$$ $$...$$ $$Y_M = c_{M,1}\times P_1 + c_{M,2}\times P_2+..c_{M,i} \times P_i+...c_{M,N}\times P_N$$

$$P$$ values are pixels and $$c$$ values are some coefficients (actually those masks in single pixel camera). So if $$N$$ is 1000, the size, $$M$$, of "measured" vector (as in CS literature), would be something around 100, that makes us, end up with a system of equation that is called under-determined. This constitute following, if we consider all $$c$$ as a matrix $$C$$ multiplied in signal $$X$$:

$$Y_M = C_{M,N}\times X_N$$

Now, if $$M>=N$$, the things will be very easy, you have a system of equations with $$N$$ unknowns and $$M$$ equations, easily solved (just multiply both sides with inverse or pseudo-inverse of $$C$$). Now the power of compressive sensing is that it reconstructs signal even in case $$M << N$$. The assumption however is the signal is sparse, i.e. 99% of elements of $$X$$ are zero (or very close to zero). This is not true if you look at an image, unless you are in dark room rarely you see any pixel of an image to be zero, however there is another trick. If you take Fourier or wavelet transform of image, you will be supersize how many elements are very small (or zero) compare to just a handful of very large elements. Hence, we say images are almost sparse in transform domain. Now, if $$X$$ is our image, $$\alpha$$ will be our transformed image as following and $$\alpha$$ will be really sparse: $$\alpha = \psi \times X$$ so obviosly, $$X$$ would be equal to $$\psi^{-1}\times \alpha$$. OK, if we substitute for $$X$$ we will have: $$Y_M = C_{M,N}\times \psi^{-1}\times \alpha$$

Single Pixel Camera

Now, we have system of equations with $$N$$ unknowns but many of those $$N$$ unknowns are zero. Therefore we can use compressive sensing to reconstruct $$\alpha$$ and consequently $$X$$.

Now, how to multiply the pixel values with $$C$$? The way single pixel camera deals is they use micro mirrors. In our 1D example we need $$N$$ micro-mirrors, which can be selected to reflect light or not as if we are multiplying the light rays with either "1" (for reflection) or "0" (for no reflection). Reflecting and focusing all rays back on our single sensor is the addition operation. Each time we set some mirrors to reflect back and focus on a single sensor and measure the light, we take a measurement, i.e. an $$Y_i$$. After the "Measurement" which is mentioned sum of production, is digitalized by our sensor read-out we can carry out rest in the computer and run compressive sensing recovery, etc, but the first part all are analog.

The benefit of course is far fewer measurements $$M< which naturally leads to more good stuff.

Now, how compressive sensing recovery algorithms solve these type of underdetermined sparse system of equations is another story. But this is pretty much how single pixel camera works.

Simulation Simply make the following process in MATLAB or Python:

1. Make $$M$$ number of binary 2D arrays with size of your image, these will be your masks

2. Calculated and sum up element-wise multiplication of each mask (ith mask) with image and call it $$y_i$$

3. Stack up the masks into a 3D array and also the $$y$$s into a 1D vector

4. Reconstruct using CS (for this you need to use many available CS recovery algorithms)

There is an example code here which might help (Disclaimer: I have not checked if it works)

https://github.com/kurokuman/single-pixel-camera-simulation