Intelligent Noise Removal in Data

Question

This question is linked to this question:

I have this data:

  data = [1.0, 0.35671858559485703, 0.44709399319470694, 0.29438948200831194, 0.5163825635166547, 0.3036363865322419, 0.34031782308777747, 0.2869558046065574, 0.28190537831716, 0.2807516154537239, 0.34320479518313507, 0.21117275536958913, 0.30304626765388043, 0.4972542099530442, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.18200891715227194, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.28830608331168983, 0.057156776746163526, 0.043418555819326035, 0.022527521866967784, 0.035414574439784685, 0.062273775107322626, 0.04569227783752021, 0.04978915781132807, 0.0599089458581528, 0.05692515997545401, 0.05884619933405206, 0.0809943356922021, 0.07466587894671428, 0.08548458657792352, 0.049216679971411645, 0.04742180324984401, 0.05822208549398862, 0.03465282733964001, 0.014005094192867372, 0.052004161876744344, 0.061297263734617496, 0.01867087951563289, 0.01390993522118277, 0.021515814095838564, 0.025260618727204275, 0.0157022555745128, 0.041999490119172936, 0.0441231248537558, 0.03079711140612242, 0.04177946154195037, 0.047476050325192885, 0.05087930020034335, 0.03889899267688956, 0.02114033158686702, 0.026726959895528927, 0.04623461918879543, 0.05426474524591766, 0.04421866212189775, 0.041911901968304605, 0.019982199103543322, 0.026520396430805435, 0.03952286472888431, 0.03842652984978244, 0.02779682035551695, 0.02043518392128019, 0.07706934170969436]

Now the question is: How can I remove the noise (colored) intelligently without removing the peaks ?

plt.plot(data)
plt.show()

Answer 1

Here is a rough cut. You could definitely get more elaborate on your peak testing, e.g. comparing it to neighbor peaks. This simply makes a metric for how far the peak "sticks out" and uses a cutoff value on that and the height to select a candidate peak.

In the case of two or more isolated peaks in your field of zeroes, it would select all of them. Not sure that is what you want, but this works for your given example.

Once your peaks are selected, then you can zero all other data values.

Ced

        thePeak_Spot       = np.zeros(100, dtype=int )
        thePeak_Height     = np.zeros(100)
        thePeak_Protrusion = np.zeros(100)

        thePeakCount = 0

        for n in range( 1, len( data ) - 1 ) :

            theLeftDrop  = data[n] - data[n-1]
            theRightDrop = data[n] - data[n+1]

            if theLeftDrop > 0 and theRightDrop > 0 :
                thePeak_Spot[thePeakCount] = n
                thePeak_Height[thePeakCount] = data[n]
                thePeak_Protrusion[thePeakCount] = (theLeftDrop + theRightDrop) / ( 2.0 * data[n] )
                thePeakCount += 1

        for p in range( thePeakCount ) :
            if thePeak_Protrusion[p] > 0.5 and thePeak_Height[p] > 0.1 :
               print p, thePeak_Spot[p], thePeak_Height[p], thePeak_Protrusion[p]  

        plt.plot(data)
        plt.show()

output:

4 13 0.497254209953 0.695280339927
5 58 0.182008917152 1.0
6 122 0.288306083312 0.900874833979