How to filter noise from continuous path in image

Question

Problem statement

In the dataset above, you can notice a continuous path and then a noise-band at around 320-400 on the x-axis. I want to be able to apply a filter that amplifies the continuous path while suppressing the noise-band. The data inside the noise-band is useless.

Attempt 1

I am new to signal processing, but I took a shot at designing my own filter. I tried designing a minimum filter. I defined it to make the output signal $ S $ the minimum of the next $ w $ terms in the input signal $ F $: $ S[n] = \min \{F[x] | x \in \{n, n+1, n+2, ..., n+w\}\} $

for i in range(len(original_signal) - window_len):
    filtered_signal[i:i+window_len] = min(original_signal[i:i+window_len])

It represents each pixel by the lowest pixel in the area which should make the noise go away because the noise band will have a lot of low values in between a lot of high values. It takes out too much of the signal as well.

This is an image showing what the filter took away (the difference between the original signal and the filtered signal)

Attempt 2

A median filter will not work, because the median of the noise-band is relatively high (1e4):

whereas the median of empty space is not that high (1e3):

So the median filter would not reduce the noise to be even with empty space.

Do you know any filters I could try?

Additional Data

Histogram of desired signal:

All histograms normalized and put on the same axes:

Answer 1

So entropy filter sounded good, but it didn't really work because the actual signal is also considered high entropy, even after adjusting the window.

What I did was take a derivative of all of the data in the time axis (left to right). Then I capped the maximum value. Then I integrated the derivative.

deriv = data[i, 1:] - data[i, :-1]
deriv[deriv > max_] = max_

The derivitave is $ \frac{dy}{dx }$. In my mind dy is a vertical line segment, dx is a horizontal line segment. Putting them together makes a right triangle, where the function is the hypotenuse is f, the horizontal leg is dx and the vertical leg is dy. Thus $ tan^{-1} {\frac{dy}{dx}} $ is the angle of elevation for f. In my mind, putting a maximum on the derivative is like putting a maximum angle on f. It is like saying, "Cap the angle of elevation for f. If it goes above 150 degrees from the horizon, make it smaller."

It seems to work. It limits the maximum angle of elevation for the signal. You can see the original signal (in red) and the signal after derivative filter (in green).

Then I take original signal minus derivative filter applied to original signal. This leaves me with just the noise band, which I can amplify, blur, and then subtract out.

Answer 2

Disclaimer: I'm wildly speculating.

I would try a bilateral kind of filter. In a patch (size depending on your image size) around each pixel, evaluate the entropy. You should have a significantly higher entropy outside the noise band, so you could simply use the minimum filter on that particular pixel if the entropy was lower than some threshold.

The problem with this approach is that the naive implementation in python will be somewhat slow. You would probably have to use Cython, Numba or something similar for implementation.

Edit: Or you could simply use the entropy filter from scikit-image.

Answer 3

I don't think there's anything in the noise band to recover. I've done several horrible things to your image in GIMP, and in no case is there any thing visible that would suggest that there's any of your signal in there.

If you look, you will see that there's a sort of repeating green tinge to the noise band - from the top down you will see sections of the noise where there's a little more green. It appears at first to be connected to the signal, but if you look closer, you will see that there's a green section where there's no corresponding signal.

You might do better working from the original signal, but I don't think you are going to get anything out of the picture.