I have been stuck on this question for a while now. It has to do with vestigial sideband.
I wasn't sure if I should be dividing $H(\omega)$ graph values by 2 because only the positive side of the filter is given....
The answer that I am getting is:
We know that $VSB(t) = A_c f(t)\cos(\omega_c t) \ast h(t)$
then in frequency form I get:
$$VSB(\omega) = A_c\pi 0.5 (\delta(\omega + (\omega_c + \omega_m)) + \delta(\omega + (\omega_c - \omega_m)) + \delta(\omega - (\omega_c + \omega_m)) + \delta(\omega - (\omega_c - \omega_m))) H(\omega)$$
Now using the graph I am multiplying the appropriate $H(\omega)$ values:
$$VSB(\omega) = A_c \pi 0.5 (0.25 \delta(\omega + (\omega_c + \omega_m)) + 0.75 \delta(\omega + (\omega_c - \omega_m)) + 0.25 \delta(\omega - (\omega_c + \omega_m)) + 0.75 \delta(\omega - (\omega_c - \omega_m)))$$
where carrier wave is given by $A_c \cos(\omega_c t)$
I don't have the solution to this, so I have no idea if my answer or reasoning is correct... It would be sooooo helpful if someone could confirm/clarify