I cannot find out what does "notch" means in image processing? We have selective filters that one of the best of them is Notch filters. but what does the concept of "Notch"? why they are called "notch" filters?
In standard 1D signal processing, notch filters are filters that try to remove one specific frequency, and pass all others.
I say "try to" because this is not usually achievable, but it comes close.
Steve W. Smiths's DSP Guide has a nice plot (Figure 19.6b below) that shows the frequency response of such filters.
In image processing, it will be the same thing, but instead of a 1D filter it'll be a 2D filter probably comprised of two 1D notch filters --- one in each image direction.