1
$\begingroup$

I have developed a service for simulating hearing impairment based on an audiogram.
However, I've encountered a problem: I can only process the entire audio signal without distortions.
Unfortunately, due to insufficient RAM when performing direct FFT, I cannot handle a sufficiently long audio signal in one go. While dividing the audio into parts seems like a solution, I face distortions when doing so.
I present three scenarios on Google Disk:

  1. The original audio (original_video.webm).
  2. The entire processed audio at once (Please increase the volume if you can't hear it).
    Contains frequencies before 1_000 Hz (the desired result) (handled_video_without_chunking.webm).
  3. The same audio with a "hammering" effect, where the repeatability of that effect depends on the chunk size. (handled_video_with_chunking.webm)

My goal is to eliminate the "thumping" effect and achieve a result as if the sound was processed as a whole at once, rather than in parts.
I also attach Java code that processes the audio:

Processing the entire signal at once (This is the working code):

private float[] processWithoutChunking(AuditoryGraph auditoryGraph) {
    float[] audioSamples = convertAudioBytesToAudioSamples(this.audioBytes);
    FloatFFT_1D fft = new FloatFFT_1D(audioSamples.length);
    fft.realForward(audioSamples);

    for (int i = 0; i < audioSamples.length; i++) {
        float frequency = (float) i * sampleRate / audioSamples.length;
        AuditoryPoint point = auditoryGraph.getPointAt(frequency);
        float attenuation = point.getAttenuation();
        audioSamples[i] *= attenuation;
    }

    fft.realInverse(audioSamples, true);
    return audioSamples;
}

Processing the signal with chunking (This code creates a hammering effect):

private float[] processWithChunking(AuditoryGraph auditoryGraph) {
    float[] audioSamples = convertAudioBytesToAudioSamples(this.audioBytes);
    final int chunkSize = (int) (sampleRate);
    List<float[]> chunkedSamples = new ArrayList<>();

    for (int i = 0; i < audioSamples.length; i += chunkSize) {
        float[] chunk = new float[Math.min(chunkSize, audioSamples.length - i)];
        System.arraycopy(audioSamples, i, chunk, 0, chunk.length);
        chunkedSamples.add(chunk);
    }

    for (float[] chunk : chunkedSamples) {
        FloatFFT_1D fft = new FloatFFT_1D(chunk.length);
        fft.realForward(chunk);

        for (int i = 0; i < chunk.length; i++) {
            float frequency = (float) i * sampleRate / chunk.length;
            AuditoryPoint point = auditoryGraph.getPointAt(frequency);
            float attenuation = point.getAttenuation();
            chunk[i] *= attenuation;
        }

        fft.realInverse(chunk, true);
    }

    float[] handledAudioSamples = concatChunkListToArray(chunkedSamples);
    return handledAudioSamples;
}

I consulted ChatGPT for advice, and it suggested using a Hanning window with overlaps. However, ChatGPT couldn't provide a working implementation, and since I have limited understanding in audio processing, I would like to know a practical approach in words on how to achieve this and whether it's possible to generate audio without distortions.

$\endgroup$
5
  • 3
    $\begingroup$ This is time domain aliasing. Frequency domain processing is mathematically complicated especially if it's time variant (which I can't really tell). I strongly suggest staying in the time domain, if you can. If not, start with overlap-add but that still requires a time domain impulse response. $\endgroup$
    – Hilmar
    Commented Jul 6 at 19:50
  • 2
    $\begingroup$ I agree with @Hilmar to an extent, but a problem deeper than being in the frequency domain when you maybe should be in the time domain is that your technique doesn't seem to reflect the underlying mechanism of the hearing loss. You could probably get much closer with either IIR filtering or by finding a sensible impulse response and filtering with that -- but it would be even better to dig into the physiological literature and maybe the psychoacoustic literature and find out how to replicate the underlying hearing loss mechanism with fidelity. $\endgroup$
    – TimWescott
    Commented Jul 6 at 20:01
  • $\begingroup$ @TimWescott, I was mistaken; the sound was too muffled (it was audible through headphones, but not on the laptop). Raised audibility since 250 Hz to 1_000 Hz. The spectrum can be checked here academo.org/demos/spectrum-analyzer $\endgroup$
    – user186103
    Commented Jul 6 at 22:02
  • $\begingroup$ I dunno how much RAM you got, but you should have enough to do a 4K point FFT and maybe double that amount. You might need 16K words. That said, you should get really formal with the Short -Time-Fourier-Transform. That's how to process audio in segments in the frequency domain without discontinuities. $\endgroup$ Commented Jul 6 at 22:10
  • $\begingroup$ @robertbristow-johnson, If I run a project on a cheap VPS, then 512 MB of RAM for the entire system. I have a laptop with 4 GB of RAM free. If I allocate 2GB to the Java Virtual Machine, then I still crash in Java Heap Space when trying to process 10 minutes of audio. My maximum is 4-5 minutes depending on the audio resolution. Perhaps I'm doing something wrong. I can suppose that Java is a bad language for such calculations. Or I'm using a not very good library. UPD: Thanks for STFT. I will think what to do next. $\endgroup$
    – user186103
    Commented Jul 6 at 22:19

1 Answer 1

1
$\begingroup$

I have always tried to avoid Java primarily because of the JVM and definitely for signal processing applications. FFTW is the fastest non-GPU FFT library I know of, but you'll have to rewrite your application in C++. I'm sure Rust has strong FFT support by now, but I haven't looked into it extensively. You might even find that Python/NumPy provides what you need as they don't use a virtual machine either, but if speed is important, I would stick to a compiled language.

$\endgroup$
1
  • $\begingroup$ I was sorta thinking the same, but I don't really know how fast Python can run. I know it's an interpreted language, but like MATLAB, many of the utilities are compiled and it's only the calling overhead and intermediate manipulations where speed suffers. If the OP was interested I would discuss STFT with them because, especially if the application gets rewritten, they might be doing literally STFT and then we know of a few little rules we gotta keep in mind if we don't wanna introduce distortion. $\endgroup$ Commented Jul 9 at 2:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.