Consider a linear time-variant channel. The transmitted signal is $x(t)$, the channel impulse response is $h(t, \tau)$, and the received signal is $y(t)$. Then $$ y(t) = \int_{-\infty}^\infty x(\tau) h(t, \tau) d\tau = x(t) \circledast h(t, \tau), \tag{1} $$ where $\circledast$ is the convolution operation. If the multipath channel is assumed to be a bandlimited bandpass channel, which is reasonable, then $h(t, \tau)$ may be equivalently described by a complex baseband impulse response $h_b(t, \tau)$ with the input and output being the complex envelope representations of the transmitted and received signals, respectively. That is, $$ r(t) = c(t) \circledast \frac{1}{2} h_b(t, \tau), \tag{2} $$ where $c(t)$ and $r(t)$ are the complex envelopes of $x(t)$ and $y(t)$, defined as $$ x(t) = \text{Re} \{ c(t) \exp(j 2 \pi f_c t) \} \tag{3} $$ and $$ y(t) = \text{Re} \{ r(t) \exp(j 2 \pi f_c t) \}. \tag{4} $$ My question is: How to derive (2)? Thanks in advance.
The following is my attempt to derive (2):
In terms of pre-envelopes, we have $x(t) = \text{Re} [ x_+(t) ]$ and $h(t, \tau) = \text{Re} [ h_+(t, \tau) ]$, where $$ x_+(t) = x(t) + j \hat{x}(t), \tag {5} $$ where $\hat{x}(t)$ is the Hilbert transform of signal $x(t)$, and $$ h_+(t, \tau) = h(t, \tau) + j \hat{h}(t, \tau) \tag {6}, $$ where $\hat{h}(t, \tau)$ is the Hilbert transform of $h(t, \tau)$. Then $$ y(t) = \int_{-\infty}^\infty \text{Re}[x_+(\tau)] \text{Re}[h_+(t, \tau)] d\tau. \tag{7} $$
In order to continue the derivation, we need the following lemma:
Lemma 1: $$ \int_{-\infty}^\infty \text{Re}[x_+(\tau)] \text{Re}[h_+(t, \tau)] d\tau = \frac{1}{2} \text{Re} \left[ \int_{-\infty}^\infty x_+(\tau) h_+^*(t, \tau) d\tau \right]. \tag{8} $$
Proof: We prove Lemma 1 by beginning with the right hand side (RHS) of (8):
\begin{align} \frac{1}{2} \text{Re} \left[ \int_{-\infty}^\infty x_+(\tau) h_+^*(t, \tau) d\tau \right] \notag &= \frac{1}{2} \text{Re} \left\{ \int_{-\infty}^\infty [ x(\tau) + j \hat{x}(\tau) ] [ h(t, \tau) - j \hat{h}(t, \tau) ] d\tau \right\} \notag\\ &= \frac{1}{2} \int_{-\infty}^\infty \text{Re} \left\{ [ x(\tau) + j \hat{x}(\tau) ] [ h(t, \tau) - j \hat{h}(t, \tau) ] \right\} d\tau \notag\\ &= \frac{1}{2} \int_{-\infty}^\infty [ x(\tau) h(t, \tau) + \hat{x}(\tau) \hat{h}(t, \tau) ] d\tau, \tag{9} \end{align} where $$ \int_{-\infty}^\infty \hat{x}(\tau) \hat{h}(t, \tau) d\tau = \int_{-\infty}^\infty \hat{x}(\tau) [\hat{h}^*(t, \tau)]^* d\tau. \tag{10} $$ Using the Plancherel theorem, (10) becomes $$ \int_{\infty}^\infty [ - j\ \text{sgn}(f) X(f) ] \{ [ - j\ \text{sgn}(-f) H(t, -f) ]^* \}^* df, \tag{11} $$ where $X(f)$ and $H(t, f)$ are Fourier Transforms of $x(t)$ and $h(t, \tau)$, respectively. (11) can be further simplified as \begin{align} \int_{-\infty}^\infty (-j)^2 \text{sgn}(f) \text{sgn}(-f) X(f) H(t, -f) df &= \int_{-\infty}^\infty X(f) H(t, -f) df \\ &= \int_{-\infty}^\infty X(f) [ H^*(t, -f) ]^* df. \tag{12} \end{align} Using Plancherel theorem again, we have \begin{align} (12) & = \int_{-\infty}^\infty x(t) [ h^*(t, \tau) ]^* d\tau \\ & = \int_{-\infty}^\infty x(t) h(t, \tau) d\tau. \tag{13} \end{align} Thus, \begin{align} (9) & = \frac{1}{2} \cdot 2 \int_{-\infty}^\infty x(t) h(t, \tau) d\tau \\ & = \int_{-\infty}^\infty x(t) h(t, \tau) d\tau \\ & = \int_{-\infty}^\infty \text{Re}[x_+(\tau)] \text{Re}[h_+(t, \tau)] d\tau. \tag{14} \end{align} This completes the proof of Lemma 1. $\square$
Using Lemma 1, we can rewrite $y(t)$ as follows: \begin{align} y(t) & = \frac{1}{2} \text{Re} \left[ \int_{-\infty}^\infty x_+(\tau) h_+^*(t, \tau) d\tau \right] \\ & = \frac{1}{2} \text{Re} \left[ \int_{-\infty}^\infty c(\tau) \exp(j 2 \pi f_c \tau) h_b^*(t, \tau) \exp(- j 2 \pi f_c t) d\tau \right] \\ & = \frac{1}{2} \text{Re} \left[ \exp(- j 2 \pi f_c t) \int_{-\infty}^\infty c(\tau) h_b^*(t, \tau) \exp(j 2 \pi f_c \tau) d\tau \right], \tag{15} \end{align} where the second equality in (15) results from (2.139) in [1]. Then I don't know how to continue...
References
[1] S. Haykin, Communication Systems, 3rd ed. John Wiley & Sons, Inc., 1994.