The DFT in of itself is not a consistent estimator, or in other words, the variance of your transform doesn't decrease as the transform length increases.
Another way to think of it is that the DFT is a linear invertible transform, so whatever "information" is contained in the time domain is contained in the frequency domain. The segment that is 256 points is expected to be less smooth than a 128 point segment. If the longer segment has more variability, it's DFT will also have more variability.
To have a consistent estimator one can use the technique attributed to Welch, which means that you average a a set of DFT segments and that generally means averaging bin magnitudes (or magnitude squares).
Picking a correct segment length is related to how many segments you will average. The number of averages is often tied to the amount of data you can average, as well as a target bin variance.
If the data has a spectrum that varies, like a speech signal, one can average too much and obscure the dynamic nature of the data.
Window functions have a large influence on smoothness, and are implicitly related to segment length.
Without knowing what your target spectrum's purpose is, any notion of "correct length" will be subjective. Subjectivity isn't necessarily bad.