First of all, common FFT libraries (which software are you using?) have FFT implementations for a lot more lengths than powers of two -- the FFTW, which is by far the most widely used in general purpose computing (and is in use by things like Matlab, FFMPEG and so forth) for example supports any length that only has prime factors smaller than 13 or so; so unless you're doing an FFT of millions of points in each direction, you don't have to pad for performance reasons.
I assume you pad before transforming:
signal -> padding -> DFT -> Filtering -> IDFT -> ?
Now, you can just truncate your result, and will get the right thing -- the IDFT works cyclic, and thus you won't be throwing away information.
Also note that if "Filter" in your original description is a FIR filter, you are convolving your signal in frequency domain, just to transform it back --- just multiply it with the fourier transform of your Filter, and you'd have done the same in the original domain of your signal, as basic Fourier theory allows you to do.