The lower the frequency, the longer the period of oscillation. In any filter using memory (IIR or causal FIR), the filter will have to evaluate a signal for a longer period of time in order to gather information about the same number of periods of a lower frequency signal component than for a higher frequency signal component. And a filter can't filter a particular spectral component unless it has gathered enough information about that spectral component to "know" whether that particular component is in the stop-band or pass-band, which takes longer (other factors being roughly equivalent) for a lower frequency signal component.
Or you could design a linear phase FIR with the same delay for a lower frequency bandpass, but since it has gathered less periods of information about that lower frequency, it's quality of filtering (width of the transition band, etc.) will be less.