Given a signal $X$, and a channel $h$, and the received signal $Y = Filter(h,1,X);$
What's the inverse of filter, it means if I know $h$, how can I get back $X$ ?
It's not possible to recover the original signal $X$ based on $h$ perfectly !!
$Filter$ or $conv$ are usually performed in time domain, so it's similar "But not exactly" to $y = conv(X,h)$ , So in that case after you know $h$ or in other words after you estimate $h$ , you need to use any equalizer like $ZF$ or $MMSE$ to get back $X$. Of course, equalization will be performed in frequency domain, since that equation you mentioned above is equivalent to $Y = X . H$ and you go into frequency domain using $fft$.
In general, what you want is impossible!
Not every operation is reversible. In frequency domain, that becomes obvious: a system's frequency response $H(f)$ might have actual zeros at some points, and thus, no inverse filter exists. Multiplication with zero can't be undone, since no matter what was at that frequency before filtering, it's 0 afterwards and we can't tell what it was.
What you describe, by the way, is an equalizer, and I'm 100% certain you've heard of these – and have good literature!
Thus, just a very quick rundown of why what you want to do is a bad idea TM:
Since, for channels $h$ with deep fades at certain frequencies (even if not completely nulled, but say, attenuated by 50 dB), your "inverse filter" (we call that zero forcing equalizer) would have to amplify the received signal at these frequencies very much. That's very undesirable, usually, because the receiver noise at these frequencies would see the same amplification.
Thus, you'd most often try to avoid that, and use other equalizers types, like MMSE.
Often, the equalizer is one of the most complicated and computationally expensive parts of a receiver, and a system designer might go through great lengths to reduce the complexity of that. In fact, you already know one such system that tries very hard to make equalization easier: OFDM (which typically actually implements zero-forcing, but goes through a lot of effort and sacrifices a lot of spectral efficiency just to be able to do it in frequency domain).