1
$\begingroup$

I am trying to implement compressive sensing for grayscale 2D images, then reconstructing them using a multi-layer perceptron(MLP). It seems to perform well no matter how many layers I add or remove, even without activation functions which is basically a linear combination of the inputs. there must me something wrong with my implementation. this is what I'm doing:

Encoding

  1. Load image (PIL)
  2. normalize image by dividing pixel values by 255
  3. partition image into $B \times B$ blocks(patches) to get a matrix of size $P \times B \times B$ where $P$ is the number of blocks.
  4. Flatten the blocks to form a matrix of size $P \times B^2$ let's call it $X$
  5. Take M measurements from each block by getting the dot product of $\Phi$, an $M \times B^2$ Gaussian matrix, and $X^T$, the matrix containing the flattened, normalized blocks. The result is the measurement matrix $Y$ of size $P \times M$

Decoding

From what i understand, this is a sparse approximation problem. meaning my images(or blocks) are supposed to be sparse in some basis $\Psi$ in order to get an acceptable construction. I assumed the basis to be 2D-DCT. Therefore for each block, i try to reconstruct $S_i$ which is a $k$-sparse vector representing the flattened block in frequency domain. So my optimization problem becomes as follows:
$$\min|S_i|_1 s.t. |\Phi\Psi^{-1} S_i-Y_i|_2 < \mu$$ $\mu$ being a small quantity. Now this applies to per-image optimization methods. What i am trying to do is an MLP which learns a universal function $f(Y_i)$ for reconstruction. So after training, my decoding process is as follows: $$\hat{S_i} = f(Y_i)$$ following the acquisition of $\hat{S_i}$ i apply $\Psi^{-1}$ to it in order to get $\hat{X_i}$. finally, $\hat{X_i} * 255$ gives me the flattened block pixel values. reshaping each $\hat{X_i}$ into a $B \times B$ matrix then arranging them gives me the full image $X$.

Training

My dataset consists of $50,000$, $256 \times 256$ images. I just use the first batch (1000 images) since adding more data does not affect my model at all somehow. I do the aforementioned encoding process to every image then concatenate and shuffle all the blocks (i.e instead of images, i now have a matrix of flattened shuffled blocks from multiple images). My model inputs are the measurements taken from these images, of size $1000*P \times M$. To get my training labels, i apply 2D-DCT to each unflattened block before taking the measurements (right after encoding step 3 above), then flatten the sparse labels $1000*P \times B^2$, let's call the matrix $S$. Loss is calculated by taking $MSE(\hat{S}, S)$. Number of layers/nodes-per-layer do not seem to affect the model accuracy at all.

My questions

  1. A Linear(no activations) MLP with a single hidden layer performs the same or sometimes better than a Multi-layer model with ReLU activations. What might be the cause of this?
  2. In the literature, to guarantee construction $X$ has to be sparse in some domain. But i tried removing all transformations from my model and just worked in spatial domain and the model performs exactly the same. shouldn't it struggle to reconstruct the signal?

Output example

A model with 1 hidden layer having $B^2$ nodes takes the input of shape $M$. then the intermediate output is given to the output layer having $B^2$ nodes. in short, $M \rightarrow B^2 \rightarrow B^2$, no activations. training $MSE$ is $0.002$. $10$ of the blocks are reserved for validation, validation loss is also $0.002$. $B = 8$, M = $ 0.25 * B^2$. Now trying the trained model on an unseen images like "lena.bmp" $(512 \times 512)$ gives the following results compared to OMP:

N.B. The same $\Phi$ used in training is used in testing.

Any help is appreciated.

$\endgroup$
0
$\begingroup$

This is a good idea, I was pondering upon awhile ago. I drop some thoughts.

A Linear(no activations) MLP with a single hidden layer performs the same or sometimes better than a Multi-layer model with ReLU activations. What might be the cause of this?

The dimension reduction transform, i.e. taking CS measurements is a linear operation, a measurement is weighted linear combination of input "sparse" vector. Are you reconstructing the same image that is used for training the network? The results could be different if the training image and test image are drastically different.

In the literature, to guarantee construction X has to be sparse in some domain. But i tried removing all transformations from my model and just worked in spatial domain and the model performs exactly the same. shouldn't it struggle to reconstruct the signal?

There is actually no need to add that. Transformation is like any other function that network can "learn". It does not matter if you map measurement to "sampling" domain or the transform domain.

Maybe you might be interested in seeing Baraniuk's work on similar thing as well: https://patentimages.storage.googleapis.com/7d/cc/57/9f8ebb657d6fb7/US10985777.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.