Define your frequency response with $N$ complex-valued samples (real parts and imaginary parts). The first sample, $F(0)$, is associated with zero Hz. The last sample, $F(N-1)$, is associated with $(N-1)F_s/N$ Hz. Make sure the $F(m)$ frequency samples are conjugate symmetrical. Then perform the inverse DFT (IDFT) of the complex-valued $F(m)$ sequence.
If your $F(m)$ sequence is truly conjugate symmetrical then the imaginary parts of your IDFT results will be VERY small in value. (In theory the imaginary parts of your IDFT results should be zero-valued.) After performing your IDFT, ignore the imaginary parts of your IDFT results.