Timeshifting in streaming based audio processing

I am currently running into a larger problem whilst programming an audio processing software. My target is time-shifting signals in frequency domain, then transform them back to time domain and merging 2 streams into an output.

I'm windowing the incoming signal(s) to extract frames of 32 samples with an offset of 16 samples. Hence, assuming 64 input (and output) samples, a total of 4 overlapping frames is created:
Frame 1 =$$[-16...-1,0...15]$$,
Frame 2 = $$[0...31]$$,
Frame 3 = $$[16...47]$$,
Frame 4 = $$[32...63]$$.
Here $$[-16...-1]$$ are samples from a previous iteration (or zeroed out in the beginning) and the last 16 samples $$[48...63]$$ are stored for the next iteration.

Assume $$x[n]$$ is the input signal in the first frame, and it is timeshifted and added to output such that $$x[n+2]=y[m]$$. For that im adding leading and trailing zeros before shifting. However, it would shift the signal out of the scope of the output buffer. Same for frame 4 and timeshifts like $$[n-2]$$, where it shifts out of the buffer (into the next iteration).

How can I handle such shifting, considering i have fixed shift values, e.g. 6 samples?

Any hint appreciated.
The picture shows how unshifted frames are assembled in the output. • why would you do this in frequency domain? An integer sample shift is just exactly that in time domain! Sep 5 '21 at 16:00
• Other processing takes place in frequency domain. It's however somewhat irrelevant to the problem description, because in the end it means processing audio samples either way. Sep 5 '21 at 16:06
• so if you're multiplying the frequency-domain data with $e^{-j 2 \pi f \tau}$ to effect a delay of $\tau$, you gotta be careful with this. it's because everything you do with the DFT is circular and your timeshifting will be circular and there can be time-aliasing, if you don't do something about it. Sep 6 '21 at 1:03
• @robertbristow-johnson , i suppose "do something about it" means zeropadding? Sep 6 '21 at 13:08
• yeah, probably. either that or recognize the aliasing at the edges. but zero-padding is the only way i know of to completely avoid time aliasing that the FFT will otherwise put in there. Sep 6 '21 at 13:19

If you absolutely must do this in the frequency domain, you can time shift the a buffer by multiplying with $$H(k) = e^{-j2\pi\frac{kn}{N}}$$ where j is frequency, n amount of delay in samples and N the FFT length.