Consider the LTI system given by: $H(z) = 1 - \frac{1}{2}z^{-1}+\frac{3}{4}z^{-2}$
Let $x[n] = (\frac{1}{2})^nu[n]$ be the input to the system. We want to find the output for $n = 0,1,...,N_a$, using the DFT of a relevant portion of the input and samples of $H(z)$ on the unit circle, at $z = e^{j\frac{2\pi}{N_b}k}$, for $k = 0,...,Nb-1$.
Now, because $H(z)$ is FIR, the samples of $H(z)$ on the unit circle are actually the values of the DFT of length $N_b$ of $h[n]$. I can work out the appropriate length of the DFT of $x[n]$ to use and it would work and using the circular convolution property, find the actual values of the output.
What if instead of a FIR, I'm given a causal IIR such as $H(z) = \frac{1}{1-\frac{1}{2}z^{-1}}$?
Can I use the samples of this new $H(z)$ on the unit circle just as in the first case? My guess is yes, because the system is stable (because it has a pole at $\frac{1}{2}$ and it is causal) and then the Fourier transform exists, so I can obtain the samples of the Fourier transform and use them as DFT of $H(z)$, (I believe) it would be like truncating $h[n]$ and then using its DFT.
Now, what if the system was unstable and causal, for example:
$H(z) = \frac{1}{1-2z^{-1}}$?
The Fourier transform clearly doesn't exist, and I can't truncate $h[n]$