How to express one image in terms of another one

Question

I have two (black and white) images of identical size - let's say 128x128 pixels. I'm interested in expressing Im2 in terms of components of Im1 such that as opposed to specifying Im2, I can specify some values (less than number of pixels in Im2, otherwise what's the point) which, together with the knowledge of Im1 would allow me to construct an approximation of Im2. Let me demonstrate by giving you my first idea (which, spoiler alert, didn't work). In pythonish pseudocode:

# Singular Value Decomposition of Im1 and Im2
U1, S1, V1 = svd(Im1)
# Function to minimise
def fit_function(S):
    Im_fit = U1*diag(S)*V1 
    diff = Im2-Im_fit
    return sqrt(mean(diff^2)) # root-mean-square difference
# Find S which fits Im_fit to Im2
Sfit = myLittleMagicSolver(fit_fun)

In other words; use singular value decomposition on Im1 and Im2. Then find numbers other than singular values of Im1 which together with U1 and V1 matrices approximate Im2. This way I could specify only 128 values - the main diagonal of the fitted S matrix.

This, however, does not work. My fitted images are nowhere close to their target of Im2. My question is then, is that a problem someone has tackled already? I am not fussed about the SVD/PCA approach. I just want to be able to express Im2 in terms of Im1.

My intended use is to train a neural network to recognize patterns in my particular dataset where one image is an input to the network, and multiple images are the output. This, of course, makes for a pretty hefty neural net. I, therefore, want to teach the network to recognize the components of the output images in terms of the input instead.

Answer 1

I found an answer which is good-enough for me. As @Stanley Pawlukiewicz has pointed out in the comments, this is hard to do for a general case when there is little correlation between the images. I, however, want to work with real images of actual things in the real world. This means there will be a lot of low frequency components (that's why jpeg compression works). If I now formulate the problem as: $$ I = U_{s} S V_{s}' $$ where $I$ is the matrix representing the image, $U_{s}$ and $V_{s}$ are the matrices of the svd decomposition of the source image, and we are trying to find $S$. Then, the inverse problem (which is the one we actually want to solve) is: $$ S = U_{s}^{-1}I(V_{s}')^{-1} $$ This is the easy part (using Matlab, python or otherwise).

I then took a good look at how are the solutions ($S$) to this problem are distributed, and they tend to cluster around the left and upper edge of the S matrix. Therefore, I can approximate S by the subset of S centered around these values. I (slightly arbitrarily) have chosen to use a quarter of a superellipse to set the cutoff value with a nice constant to play with. I can get S down to 10% of its elements and retain most of the broad features of the image, which is all I wanted.

I've put my code up online so you can check it out. Hope this helps someone. :)