"why wouldn't you just use a small number of points and pad with zeros all the time"
The thing you have to remember is that with the DFT what you are doing is sampling the continuous discrete spectrum of your signal, which is the real thing that you want to observe and analyse. So lets say you have some signal, and you take only 4 samples, and then zero pad it with 1020 zeros, and do a DFT of 1024. Effectively what you are doing is taking your discrete time signal, applying a square window of size 4 (leaving only 4 samples), which will result in your spectrum being "smeared out" due to the operation of the convolution in the frequency domain with the transform of your size 4 square window, and then with the DFT sampling that spectrum to a very good resolution. So you get a lot of detail, to observe a spectrum which has been ruined in the first place by taking to few samples.
In fact, if you are going to implement the DFT in hardware using the FFT algorithm, zero padding is almost never done. The only reason you would zero pad is if you have to plot the spectrum and you want many points so it looks smooth. But in real time processing, since you don't actually gain any information by doing it, you would just do a DFT of the number of time samples that you have.
Always remember, if you have a discrete time signal, the "Real spectrum" is obtained with the Discrete time Fourier Transform, which is a continuous function of $\omega$. The DFT is a computational tool for calculating it with a computer.