Does the following define a linear or nonlinear system?
$$ y(n) -4 y(n)y(2n)=x(n) $$
Using the definition of linear system as one which can be put in the form:
$y_0 + a_1 y_1 + \dots + a_n y_n = b_0 x_0 + \dots + b_m x_m$
because of the multiplicative term $y(n)y(2n)$ we can conclude that the system is nonlinear.
Your equation does not really define a system, as we don't know what the input/output pair is (is $x$ the input, or $y$?), and where the values of $x$ or $y$ dwell. It is not even well-defined: at $n=0$, we have the equation $y(1-4y)=x$, which may have two, one or zero solutions.
However, let us suppose that $x$ is the input, and $y$ the output. The product of two "data" in $y[n]y[2n]$ is suspicious with respect to linearity. Because in a such a product, multiplying a variable by $\lambda$ turns out to become a product by $\lambda \times \lambda = \lambda^2$, hence a risk of loss of linearity. This is just a first intuition, now we shall try to prove it.
When you suspect that a system is "not something", a counter example suffices, and is often easier to find (at least in
So, if $x$ is multiplied by $\lambda$, what happens if we suppose that the system is linear, and $y$ is multiplied by $\lambda$ as well? We have two equations:
Can they be fulfilled together? If both terms on the first equation do not vanish, by division, the second one yields $\lambda -4\lambda^2 = \lambda$, meaning that $\lambda=0$. So, it is not linear (whenever $x[n] \neq0$).