Short answer; I'll try to come back and improve this later.
Recall that a matched filter is equivalent to a sliding correlator. At each time instant, the sliding correlator multiplies the last $T$ (where $T$ is the pulse duration) of the input signal by the conjugate of the pulse shape and coherently integrates the result over the pulse duration. If there is no frequency offset, then the result after multiplying by the conjugated pulse shape is a constant (DC) value, so you don't observe any loss.
If there is a frequency error, then the result after multiplying by the conjugate of the pulse shape is a sinusoid whose frequency is commensurate with the frequency offset. The scalloping loss is incurred is due to the fact that you're integrating the value of said sinusoid over a time window of length $T$. Therefore, the amount of attenuation can be well-modeled by the frequency response of a boxcar filter of length $T$:
$$
\text{rect}\left(\frac{t}{T}\right) \Leftrightarrow T \text{ sinc}\left(\frac{2\pi f T}{2}\right)
$$
You can make the following observations from this relationship:
As the pulse duration $T$ increases, the frequency response of the correlator rolls off more quickly; for a given frequency offset $f$, you'll experience more attenuation from the ideal peak value.
As the frequency offset $f$ increases, the frequency response of the correlator rolls off more quickly as well.
These combine to tell you what you probably already intuitively know; the longer you coherently integrate a signal (i.e. the longer the pulse duration in your case), the more sensitive the result will be to frequency offset.