It's true that the algebraic expression for the $\mathcal{Z}$-transform is generally not sufficient for computing the corresponding time-domain sequence. The additional information we need is the region of convergence (ROC). The ROC together with the expression for the $\mathcal{Z}$-transform uniquely determines the corresponding time-domain sequence.
For the given example
$$H(z)=\frac{1}{1-az^{-1}}\tag{1}$$
it is clear that the expression $(1)$ can be obtained in two ways. First,
$$\begin{align}-\sum_{n=-\infty}^{-1}a^nz^{-n}&=-\sum_{n=1}^{\infty}a^{-n}z^n\\&=-\frac{\frac{z}{a}}{1-\frac{z}{a}}\\&=\frac{1}{1-az^{-1}},\quad |z|<|a|\end{align}$$
corresponding to the sequence $h[n]=-a^nu[-n-1]$, and, secondly,
$$\sum_{n=0}^{\infty}a^nz^{-n}=\frac{1}{1-az^{-1}},\quad |z|>|a|$$
corresponding to $h[n]=a^nu[n]$. In both cases I've used the formula for the geometric series.
The only difference is the ROC, so it's crucial to know the ROC in addition to the expression for the $\mathcal{Z}$-transform. The ROC $|z|>|a|$ indicates a right-sided sequence, whereas the ROC $|z|<|a|$ corresponds to a left-sided sequence. For higher order systems, you can also have an annulus $|a|<|z|<|b|$ as ROC, corresponding to a two-sided sequence.
If the sequence is interpreted as the impulse response of a system, then the system is stable if the ROC contains the unit circle $|z|=1$. It is causal if the sequence is zero for $n<0$.