The short answer is do not null out the "mirror frequencies" that are located above $f_s/2$ that match the frequencies you want to keep. If your FFT was generated from a real signal, then when you do the IFFT you will get the real signal back as long as you did not zero out those upper frequencies (as you did).
The DFT (which the FFT computes) returns complex values but for real signals the “positive” frequency values will be the complex conjugate of the negative frequency values which together represent real signals. Due to how the FFT is computed, instead of returning the positive and negative frequencies from $-f_s/2$ to $+f_s/2$, instead those same frequencies are given from $0$ to $f_s$.
Consider Euler’s Identity:
$$2cos(\omega t) = e^{j\omega t}+ e^{-j\omega t}$$
If it wasn’t already clear, the general form of $Ae^{j\theta}$ represents the magnitude and phase of a complex number as $A\angle{\theta}$ so above we see how the cosine is represented by two phasors each rotating in time in opposite directions (a positive and negative frequency) such that their sum is always on the real axis.
The FFT returns the frequency bins from 0 to one sample less than the sampling frequency: $n =0$ to $N-1$ bins where each bin is spaced by $f_s/N$ with $f_s$ as the sampling rate. Due to the cyclical nature of the FFT this represents the positive and negative frequencies by mapping everything above $f_s/2$ to the negative frequencies, specifically samples $N/2 + n$ for $n\ge N/2$ map to $-N/2+n$. The command fftshift in MATLAB or Octave does this specifically.
For this reason you should NOT null out all the "mirror" frequencies that are located above $N/2$ as without them your signal will be complex. Consider the example above with the cosine.
$$2cos(2\omega_n t) = e^{j2\omega_n t}+ e^{-j2\omega_n t}$$
If sampled such that there were 100 total bins, and $\omega_n$ was the spacing on one DFT bin, then the DFT would have non zero values at the 3rd bin (bin 0 is DC) and the 2nd to last DFT bin which after shifting would be at the expected +/-2 bins. If you removed the higher bin then the resulting waveform would simply be:
$$e^{j2\omega_n t}$$