Now is there a general method to analyze and determine the exact function of that effect
No, there can't be. Effects can (and will be) arbitrary, non-linear, memory-affected mappings.
You will have to make a model of what the effect does and then look for an estimator for the parameters of that model.
For example, if you radically restrict yourselft to LTI systems (e.g. linear filters), then you can represent the effect by its impulse response, which is easy to observe using an impulse.
But, as said, the effects that are specific to expensive DAW will at least have memory and often quite a bit of nonlinear behaviour. Your model becomes very complex then, and there's no guarantee any model matches what your effect actually does!
Really, effects are arbitrarily complex computer programs, if you look at it this way. You want to figure out what a program does by observing its output. Can you tell me the source code of an encryption program when I give you the plaintext and key and ciphertext? How many sets of inputs and outputs would you need to be able to do that in general?
cool audio effect that works well on guitar, sadly it exists only in the DAW environment
Don't underestimate the computational load of effects; many will simply be too hard for anything that's not as powerful as a PC. Modern effect boxes that can digitally emulate a wide range of analog distortions as well as apply purely digital effects use pretty beefy specialized digital signal processors, and/or FPGAs to even be able to compute these things.
Also, and that's not even solvable by throwing more compute power at the problem: A lot of post-processing algorithms simply aren't causal. That simply makes it impossible to apply live.
would using Audio Neural Style Transfer be the best option
"Best" is never defined unless you give an objective measure for goodness. However, no:
Style transfer is style transfer. You want to estimate the properties of a system; that's a different problem.
But neural networks/ machine learning in general:
Yes! The central statement of neural networks is the universal approximation theorem, which says that given enough nonlinearly weighted linear projections, you can approximate any function.
I'm currently working with a student on training neural networks to learn how the nonlinear distortion of a high-power amplifier looks, so that he could use that model to train a predistorting network to counter these effects. It's working well, but the model we're using now is memoryless and relatively benign in nonlinear distortion. Learning an arbitrary guitar effect would, on our high-end GPUs, take months to learn (to the degree that satisfies our requirements).
Whenever you can, start with a known model rather then letting your machine learn from scratch. It shows that really, it's way quicker converging if you can offer a preprocessed version (that might not perfectly match the desired function). And you have a really hard problem: While my student has the luxury of not having to assume memory, you necessarily have to consider previous input. That makes the dimensionality explode.
