Suppose we have filter $h[n]$, length $N$. If I use it to do convolution in the time domain, then there will be $N$ multiplications each time during the sliding. (Ignore border conditions for now). However if say $h[n]$ had zeros every other sample, then there would be half that number of multiplications, so ostensibly I save in computational load, since I have half the number of multiplies to do.
So then in conclusion the speed of convolution in the time domain is related to the number of zeros.
Now suppose I did this convolution via the multiplication in the frequency domain using the FFT instead. It seems that I would undo any gains I could have had, zeros or no zeros, because when we do it via the FFT method, it is agnostic to the actual values of the samples of the signal and filter.
Is this a correct assessment? If so, might there then be cases where a good old fashioned convolution in the time domain is in fact much faster than an FFT?