I'm trying to find the frequency response $$H(\omega) = Y(\omega)/X(\omega)$$ for this system- the signal equations are given: $$y[n] = v[n - M] - g * v[n]$$ $$v[n] = x[n] + g * v[n - M]$$
I've tried using the z transform, and the fourier transform, but I just don't know how to get rid of the recursive terms to get it in terms of $Y(\omega)$ and $X(\omega)$.
The system is illustrated as follows:
I'll go through it in the $z$-domain. First, we find the transfer function $H_1(z) = \frac{V(z)}{X(z)}$. As you noted, in the time domain, $x[n]$ and $v[n]$ are related as follows:
$$ v[n] = x[n] + g * v[n - M] $$
Take the $z$-transform of the above and you get:
$$ V(z) = X(z) + z^{-M}G(z) V(z) $$
taking advantage of the convolution property, which states that $x_1[n] * x_2[n] \Leftrightarrow X_1(z)X_2(z)$, as well as the time-delay property, which states that $x[n-M] \Leftrightarrow z^{-M}X(z)$. Now, we can rearrange terms in the above to get:
$$ H_1(z) = \frac{V(z)}{X(z)} = \frac{1}{1 - z^{-M}G(z)} $$
Now, find the transfer function $H_2(z) = \frac{Y(z)}{V(z)}$. They are related in the time domain as follows:
$$ y[n] = v[n-M] - g * v[n] $$
Take the $z$-transform of the above to yield:
$$ Y(z) = z^{-M} V(z) - G(z)V(z) $$
Rearrange terms again to get:
$$ H_2(z) = \frac{Y(z)}{V(z)} = z^{-M} - G(z) $$
Now, you can get the overall transfer function $H(z)$ by multiplying the two:
$$ \begin{align} H(z) &= H_1(z) H_2(z) = \frac{V(z)}{X(z)}\frac{Y(z)}{V(z)} = \frac{Y(z)}{X(z)} \\ &= \frac{1}{1-z^{-M}G(z)} \left(z^{-M} - G(z)\right) \\ &= \frac{-G(z) + z^{-M}}{1-z^{-M}G(z)} \end{align} $$
which has some nice symmetry to it. The overall response will be dependent upon the system $g$ in the feedback path. If you want to evaluate the frequency response, just let $z = e^{j\omega}$:
$$ H(\omega) = \frac{-G(\omega) + e^{-j\omega M}}{1-e^{-j\omega M}G(\omega)} $$
As a side note: this contraption is known as a Schroeder-Allpass (see for example https://ccrma.stanford.edu/~jos/pasp/Schroeder_Allpass_Sections.html) and often used in creating artificial reverb. For M=1 (i.e. a single sample delay), it degenerates into a regular first order all pass.
Replacing the single tab with a multi-tab delay still maintains the overall all pass shape but with a phase that's "compressed" and "repeated" and hence a much larger group delays. This is actually true for any all pass: if you replace all delays inside an all pass structure with different delays or even all passes, the whole contraption stays an allpass. That is simply a function of the Z transform.
