Multiplications of time domain vectors of two signals is equivalent to convolutions in frequency domain and also use circular convolutions for when FIR filter design using window method in frequency. if working Discrete Fractional Fourier transform (DFRFT) i.e. on bins between time and frequency domain then which is proper operation (multiplication, convolution or any other operational treatment) gives same effect for FIR filter design using window when DFRFT of window function and FFT of Ideal filter is taken? Please explain with example if possible.
The question is very unclear. Whether convolution is circular or not depends only on what flavor Fourier Transform you use. It has nothing to do with filter design. It has also nothing to do with whether signal are "on bins" or not.