This is from Klapuri 2003. What is the "frequency response" and why do we model is as multiplied with the vibrating system we want to get the frequency from?
Taking a periodic function with many harmonics, and then filtering it, is a common way of generating (or modeling) periodic signals.
It is how you can model human speech, where S is the sound produced by vocal cords, and H is the frequency response of the cavities formed by larynx, mouth, lungs, etc. And this is how many vocoders analyze and synthesize signals.
Imagine the signal S to be a periodic signal with all harmonics; ideally, S may be an impulse train, so that you have harmonics present at all integer multiples of the fundamental frequency.
Then, H is a filter which will affect each harmonic independently, in order to produce (or model) the desired periodic signal.
Since the variable $k$ corresponds to frequency, we can assume that they are actually DFT coefficients. Applying a filter H to a signal S is a multiplication in frequency (or a convolution in time).
Frequency response of system usually take form of chart where x-axis represent frequencies, and y-axis the attenuation or response of system. For example here is a frequency response of microphone:
From this graph we can see that frequencies below 200Hz get attenuated. When picking up a sound of 50Hz you will get -10dB weaker signal as compared to 200Hz sound source of the same amplitude. Loss of -10dB means that the power is 10 times smaller or amplitude about three times smaller. At 6 kHz you would get +5dB stronger output (3.17x power or 1.77x amplitude)
Following image shows frequency dependency of gain of simple band pass filter. Many times you are also interested how the phase of input signal gets distorted, this can be seen in the second graph - frequency dependency of phase shift
When you are feeding a system with some test signal, you just multiply it's spectrum with frequency response of a system to get the resulting spectrum at system's output. Also the system introduces some noise and this can be expressed in the form of equation from your book.