I'm an undergraduate electrical engineering student who has an interest in mathematics of signal processing. I want to be able to go to a graduate school where I can focus on more the mathematics and theory behind signal processing. Can this be achieved in a Masters in EE or is it more efficient to go to my Masters in APPLIED MATH?
Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.
As stated in the comments, this is more about the content than the label. The best is to play with both fields if you want to invest in the future: signal processing is borrowing increasing from mathematical tools, for instance convex and non-convex optimization, learning, topological and geometrical tools (for dimension reduction), graphs, etc., that might not be taught in traditional EE. Such techniques can help with the growing quantity of data with little models and knowledge.
As a personal anecdote, I attended a lecture on "math for EE engineers" for the beginning of a signal processing PhD program in a US university. It was about Euclidean vector spaces (in finite dimension of course). A young PhD student asked whether the basis vectors did belong to the space. I was quite surprised by the level of the question. But not as much as when I hear the teacher say, after a long pause: "mmmm, I'd say... yes, let me think about it" (such a question is indeed interesting in infinite dimension spaces). There are similar stories from colleagues attending other MsC programs.
So after picking the right topics, I strongly suggest that you either complement an EE program with maths, to get better insight and intuition about the theoretical tools, or dive in maths, while practicing a lot on "real DSP problems" aside. The gap is large between mastering Fourier theory and being able to properly estimate and interpret of a power spectrum of a noisy signal. There are plenty of online resources to assist you in that task.
I really appreciate the concept behind the law of excessive learning in mathematics, stated by Alexander Borovik:
To be able to use maths at certain level it is necessary to learn it at the next level