I am trying to compute a compression filter function to minimize the error from a a desired pulse compression result.
The usual approach to doing this is to minimize the RMS error of the the compressed signal while it's represented in linear terms (i.e., in volts). This method handles complex signals and pulse compression for my nonlinear FM signal, and appears to work as advertised. (The method is described in the end notes of this question.)
While the method minimizes the RMS error, it leaves some individual samples larger than I'd like them to be, while producing some that are much lower than they need to be. What I'd like is a time sidelobe response that is more uniform in the time domain. Something like the sidelobes of a Chebyshev Type II filter function is what I'm looking for, but two-sided and in the time domain, not the frequency domain:
This would be much more useful for my purpose (which is for a radar application).
My first thought is to perform the RMS calculations in decibels, rather than as linear values. If I could do that, the same optimization function can be used to solve the liner equations for minimizing the error. It may work or it might not, but that's all I can think of at the moment. One problem is that I don't know how I would handle the complex signals and the complex compression function if I do this in decibels. So I'm stuck at the moment.
Can someone give my some guidelines or a solution to achieve what I'm looking for?
End Notes:
Here is the general method for minimizing the RMS error:
Given a complex pulse $$x(n), \ \ 0 \leq n \lt N$$ and a pulse compression function $$\omega(k), \ \ 0 \leq k \lt K$$ we have the compressed pulse $$z(n)=\sum_{k=0}^{K-1}x(n-k)\omega(k), \ \ 0\leq n \lt (N+K-2)$$
If the desired compression results is $$d(n), \ \ 0\leq n \lt (N+K-2)$$ and the error is defined as $$\epsilon(n)=z(n)-d(n)$$ the residues is deifned as $$\epsilon^2=\sum_{n=0}^{N+K-2} |d(n)-z(n))|^2 = \sum_{n=0}^{N+K-2} |d(n)-\sum_{k=0}^{K-1}x(n-k)\omega(k)|^2$$
If $Z$ and $D$ are column vectors for $z()$ and $d()$, we can write $$\epsilon^2=(D-Z)^{*}\cdot(D-Z)= D^*D-D^*Z-Z^*D+Z^*Z$$
$Z$ can be written as $$Z=\begin{bmatrix} X & 0 & 0 & 0 & 0 \\ 0 & X & \vdots & 0 & 0 \\ X & 0 & \vdots & 0 & 0 \\ 0 & 0 & \vdots & X & 0 \\ 0 & 0 & 0 & 0 & X \end{bmatrix}\cdot W = M \cdot W$$ $X$ and $W$ are column vectors for $x()$ and $\omega()$ and $M$ is a $(N+K-1)$ by $K$ matrix.
I'll omit the intermediary steps, and conclude that $\epsilon^2$ is minimized with respect to $w$ when $$W = (M^{*} \cdot M)^{-1} \cdot M^{*} \cdot D$$
FOLLOW UP Not sure if this is the best place to put this, but the recommendation from Matt L. to use iterative reweighted least squares did produce some improvement.
Using a weight vector $p(n)$ (I already am using $w$ and $W$), this converts to the following solution in matrix form: $$W = (M^{*} \cdot P \cdot M)^{-1} \cdot M^{*} \cdot P \cdot D$$
And here is an example of the result for a specific case, using the matched filter, the first iteration, and the 50th iteration. What I see is great improvement in the lobes I wanted to reduce. But I had hoped it would level out the sidelobes to something more even. A few more iterations led to a singular matrix problem, so I didn't go any further.