Loss of Resolution When Shifting Up With FFT / Phase Vocoder Pitch Shifter (Compared to Shifting Down)

Question

I am working on a project writing a realtime FFT / phase vocoder based pitch shifting algorithm in C++. In development, I notice that shifting down has much better audible pitch tracking and resolution than shifting up, using the same overlap and bin settings. In particular I notice pitch up starts sounding very AM / ring modulated at certain harmonies, particularly the minor 3rd.

Here is my pitch shifting function for reference:

void pv_process(void){
    memset(pv_out, 0, fft_size * sizeof(float32_t));
    float32_t shift = pitch_shift.get();            
    for(int i = 2; i < fft_size; i += 2){
        int n = i / 2;
        int new_bin = int((float32_t)n * shift + 0.5f) * 2;
        new_bins[n] = new_bin;
        if(new_bin > 0 && new_bin < fft_size){
            pv_out[new_bin+1] = pv_in[i+1] * shift;
            pv_out[new_bin] += pv_in[i];
        }
    }
}

The process works pretty simply, by transposing frequency and amplitude values to a new bin based on the pitch shift ratio, and multiplying the pitch value by the same ratio (result is a higher / lower pitch in a higher / lower bin with the same amplitude).

What I realized was that both cases reduce the number of bins containing information. A shift down of one octave (0.5 shift) will have half the number of input bins filled, running from zero to half of the nyquist frequency. Meanwhile a shift of one octave up will have half the number of input bins filled, but those will occupy every other bin running up to the nyquist frequency. In either case I'm essentially reducing the FFT size by 2, but in the case of a shift down it will all be in the lower range where consonant harmonic information lives, whereas with a shift up half of this range will be empty bins.

Does this explanation make sense for the effect I'm hearing? And how can I account for this effect to improve the sound quality?