Given $y[n]=h_1[n]x[n]+h_2[n]x[n-1]+h_3[n]x[n-2]$ is an LTI system with unity gain at $\omega =0$ and zero gain at $\omega =\pi$. $h[n]\neq0$ .Also given that the system has a linear phase. Compute
$h_1[n]$
$h_2[n]$
$h_3[n]$
How do we approach a question like this? Any help is appreciated. I want to confirm my working for $h_1[n]$ and I found other two co-eficients as 1/4
You're overcomplicating things here. There's no need for sines and cosines and squares. Note that $\omega=0$ corresponds to $z=1$, and $\omega=\pi$ corresponds to $z=-1$. From the definition of the $\mathcal{Z}$-transform you should be able to figure out that the DC term of the transfer function $H(z)$ is given by
$$H(1)=h_1+h_2+h_3\tag{1}$$
and the value at Nyquist ($\omega=\pi$) is
$$H(-1)=h_1-h_2+h_3\tag{2}$$
Equating $(1)$ and $(2)$ with the desired values at these two frequencies gives you two equations for three unknowns. The third equation comes from the linear phase requirement. I'm sure you can take it from here.
