Your formula for the RMS bandwidth makes sense for (perfectly band-limited) *low pass* signals, i.e., for signals with a spectrum centered around $f_0=0$. The bandwidth of low pass signals is defined as the support of their spectrum at *positive* frequencies. So your integration limits must be $-B$ and $B$. This results in

$$B^2_{rms}=\frac{\int_{-B}^{B}f^2df}{\int_{-B}^Bdf}=\frac{2B^3/3}{2B}=\frac{B^2}{3}\tag{1}$$

which is the expression that you're looking for.

Note that the RMS bandwidth is also defined for signals that are not ideally band-limited. In that case you have to integrate from $-\infty$ to $\infty$, as pointed out in [MBaz's answer](http://dsp.stackexchange.com/a/33707/4298).

If you have a (not necessarily perfectly band-limited) band pass signal centered around $f_0\neq 0$, you must use the following formula for the RMS bandwidth:

$$B^2_{rms}=\frac{\int_{0}^{\infty}|H(f)|^2(f-f_0)^2df}{\int_{0}^{\infty}|H(f)|^2df}\tag{2}$$

If $|H(f)|$ is perfectly band-limited and if it is constant ($|H(f)|=c$) in the interval $[f_0-B/2,f_0+B/2]$, its RMS bandwidth is

$$B^2_{rms}=\frac{c^2\int_{f_0-B/2}^{f_0+B/2}(f-f_0)^2df}{c^2\int_{f_0-B/2}^{f_0+B/2}df}=\frac{c^2\int_{-B/2}^{B/2}f^2df}{c^2B}=\frac{B^2}{12}\tag{3}$$

which equals your original (wrong) result for the low pass case. This is no surprise because a low pass signal with bandwidth $B/2$ becomes a band pass signal with bandwidth $B$ when modulated up to center frequency $f_0$.