1
$\begingroup$

I wrote some golang code which implements 3Blue1Brown's Hilbert's Curve: Is infinite math useful? ... give the code an image and it returns back its audio equivalent ( inverse Fourier Transform ) ... also wrote the same algo in reverse ... give it audio and it returns the image representation ( Fourier Transform )

This works perfectly ... too perfect ... the audio is both produced and consumed by machine however it is not human listenable ... audio in time domain

This plot show the output audio in time domain which I need to perform some post processing on to make human listenable ... notice the initial monster peak followed by relative silence ... as per the linked-to 3Blue1Brown video the algo visits each input image pixel giving first pixel the lowest audio frequency and the last pixel the highest frequency, each pixel inbetween gets a freq inbetween ... output audio is the IFFT of the input image

Does anyone know of an approach to somehow morph my audio to make it human listenable ? right now the audio sounds like a pop followed by silence for one second

Here is an input image Here is an input image which is processed and transformed into this audio here is the output audio which started life as an image for completeness here is the image synthesized from that audio image synthesized from that audio

The output audio is from an IFFT call https://github.com/mjibson/go-dsp/ so my guess is that audio initial spike quickly followed by a greatly attenuated signal happens due to most of the audio frequency oscillators all working in unison as their periodicity work to positively reinforce

Approach I have tried (naturally these put the kibosh on machine consumption fidelity)

moderate audio amplitude so it gently flattens out not the current abrupt ( think forcing the hand by semi normalizing the signal )

I really want to get this working as I intend to ship this so blind folk can hear the audio of whatever they point their phone camera at

PS yes my implementation straps on the Hilbert Curve to determine pixel traversal as per the inspirational 3blue1brown yt video

$\endgroup$

1 Answer 1

2
$\begingroup$

You may have noticed that the video doesn't show a single example of any audio that's been generated this way. I think the key quote here is

Let's temporarily set aside worries about whether or not this actually would work

The main problem that you run into here is that the spectrum of an audio signal (or any signal) is complex (i.e $\in \mathbb{C}$). The algorithm as described determines the magnitude of the spectrum at a any given frequency but not the phase. In your case you have assumed the phase to be zero and hence everything bunches up around $t=0$ and all you get is a single pop. If you were to choose a random phase instead, you would get a continuous noise signal. These sound completely different despite having exactly the same magnitude spectrum.

There are a lot of other problems you'd be likely to run into: Human perception of frequency is logarithmic whereas the algorithm assumes a linear mapping of pixel location to frequency. A very important feature of sound is repeating patterns which translates into tonality an pitch: its not clear (to me) what feature in the picture would translate into that type of audio perception.

I think this is a great idea and a super meaningful application. But I'm not sure that this approach is viable, at least not without some major modifications. You may need a mapping functions that translates important visual properties onto important auditory properties.

$\endgroup$
1
  • $\begingroup$ I need to revisit using a random phase shift offset per frequency ( per pixel ) ... it did make the audio listenable ... that was when I used a home made inverse Fourier Transform now using a git repo for FFT and IFFT $\endgroup$ Commented May 23, 2023 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.