estimating spectral optimization

I'm relatively new to DSP so excuse my simplified words, and my detailed explanation.

if the signal have non-coherent sinusiod, it will induce energy spreading into the frequency domain.

One of the solutions is :

how to estimate the signal component correctly?

And I'm stuck here, I don't really know how to optimize such problem and setting a stopping criterion in Matlab,R..etc

How to optimize such objective function?

What are you thoughts about the drawbacks about such solution to separate signal components?

• So, your signal is $x(t)=a_1\sin(2\pi f_1t)+a_2\sin(2\pi f_2 t)$ with $a_1,a_2,f_1,f_2$ unknown? What about the phase of the sines? – Maximilian Matthé May 7 '17 at 14:05