The goal is to predict a genetic distance of species from the spectra. For instance if the genetic distance of species 1 and species 2 is very height I would expect a different spectra. If the distance is 0 or very low,m the spectra are the same /very similar.
In that case, defining the distance between two spectra based on the spectra makes no sense – you want to define it based on the genetic distance.
So, you want to infer a model that describes genetic distance based on spectra. That's fine – but such a model will be very complex, and it's (as far as I can tell) not possible to find an easy function that will just give you what you want.
Unless you have a large dataset of genetic distances, I'll even go as far as saying that this is scientifically a very dangerous thing to do: your spectra are themselves large sets of data (a lot of points, say 1000). If you have, say, only 100 genetic distances, then you'll always find something that correlates well with the genetic distance within your large number of points, without having anything to do with it: the 101. genetic distance would not be represented by that model at all, i.e. it would only fit to the data set you had when you formed that model.
Basically, what you're describing is a problem that these days would usually be solved using neural networks. Again, this all depends on you having enough data to train your network, or else your network will not generalize to new spectral measurements and be useless outside your training data set.
