0
$\begingroup$

Okay so I'm relatively new to signal processing and don't know the specific terminology to help narrow my search, but I'm looking for advice in the form of books, papers, references, etc that can help me solve the following problem:

Lets say a dry audio signal is passed through a digital effect or VST in a workstation.
An effect is thus applied to the input sample generating the wet output.

Now is there a general method to analyze and determine the exact function of that effect so that we could reverse engineer it in the form of some physical I/O system we build? (ex: find a really cool audio effect that works well on guitar, sadly it exists only in the DAW environment and isnt exactly practical for live performance, reverse engineer effect it into a pedal, can now plug in any instrument.)

bonus: would using Audio Neural Style Transfer be the best option? (in which case any machine learning references that might help I'd be grateful to read!)

$\endgroup$
  • $\begingroup$ On the DAW point: The "problem" you mention is quickly becoming straightforward to deal with. VST plugins are compiled down to DLLs (or Shared Objects in Linux) with known internals. It then becomes the issue of establishing an audio "loop" and then processing sample frames. This is perfectly doable with a RPi-Zero today and even more straightforward with more powerful SBCs. There are already products and projects along these lines. System identification of an effects chain is still interesting but along the lines of MMs response (i.e. optimise for known building blocks). $\endgroup$ – A_A Feb 16 at 16:30
3
$\begingroup$

Now is there a general method to analyze and determine the exact function of that effect

No, there can't be. Effects can (and will be) arbitrary, non-linear, memory-affected mappings.

You will have to make a model of what the effect does and then look for an estimator for the parameters of that model.

For example, if you radically restrict yourselft to LTI systems (e.g. linear filters), then you can represent the effect by its impulse response, which is easy to observe using an impulse.

But, as said, the effects that are specific to expensive DAW will at least have memory and often quite a bit of nonlinear behaviour. Your model becomes very complex then, and there's no guarantee any model matches what your effect actually does!

Really, effects are arbitrarily complex computer programs, if you look at it this way. You want to figure out what a program does by observing its output. Can you tell me the source code of an encryption program when I give you the plaintext and key and ciphertext? How many sets of inputs and outputs would you need to be able to do that in general?

cool audio effect that works well on guitar, sadly it exists only in the DAW environment

Don't underestimate the computational load of effects; many will simply be too hard for anything that's not as powerful as a PC. Modern effect boxes that can digitally emulate a wide range of analog distortions as well as apply purely digital effects use pretty beefy specialized digital signal processors, and/or FPGAs to even be able to compute these things.

Also, and that's not even solvable by throwing more compute power at the problem: A lot of post-processing algorithms simply aren't causal. That simply makes it impossible to apply live.

would using Audio Neural Style Transfer be the best option

"Best" is never defined unless you give an objective measure for goodness. However, no:

Style transfer is style transfer. You want to estimate the properties of a system; that's a different problem.

But neural networks/ machine learning in general:

Yes! The central statement of neural networks is the universal approximation theorem, which says that given enough nonlinearly weighted linear projections, you can approximate any function.

I'm currently working with a student on training neural networks to learn how the nonlinear distortion of a high-power amplifier looks, so that he could use that model to train a predistorting network to counter these effects. It's working well, but the model we're using now is memoryless and relatively benign in nonlinear distortion. Learning an arbitrary guitar effect would, on our high-end GPUs, take months to learn (to the degree that satisfies our requirements).

Whenever you can, start with a known model rather then letting your machine learn from scratch. It shows that really, it's way quicker converging if you can offer a preprocessed version (that might not perfectly match the desired function). And you have a really hard problem: While my student has the luxury of not having to assume memory, you necessarily have to consider previous input. That makes the dimensionality explode.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.