Your formula for the RMS bandwidth makes sense for (perfectly band-limited) low pass signals, i.e., for signals with a spectrum centered around $f_0=0$. The bandwidth of low pass signals is defined as the support of their spectrum at positive frequencies. So your integration limits must be $-B$ and $B$. This results in
$$B^2_{\rm rms}=\frac{\displaystyle\int_{-B}^{B}f^2df}{\displaystyle\int_{-B}^Bdf}=\frac{2B^3/3}{2B}=\frac{B^2}{3}\tag{1}\\\\$$
which is the expression that you're looking for.
Note that the RMS bandwidth is also defined for signals that are not ideally band-limited. In that case you have to integrate from $-\infty$ to $\infty$, as pointed out in MBaz's answer.
If you have a (not necessarily perfectly band-limited) band pass signal centered around $f_0\neq 0$, you must use the following formula for the RMS bandwidth:
$$B^2_{\rm rms}=\frac{\displaystyle\int_{0}^{\infty}\lvert H(f)\rvert^2(f-f_0)^2df}{\displaystyle\int_{0}^{\infty}\lvert H(f)\rvert^2df}\tag{2}\\\\$$
If $\lvert H(f)\rvert$ is perfectly band-limited and if it is constant ($\lvert H(f)\rvert=c$) in the interval $[f_0-B/2,f_0+B/2]$, its RMS bandwidth is
$$B^2_{\rm rms}=\frac{c^2\displaystyle\int_{f_0-B/2}^{f_0+B/2}(f-f_0)^2df}{c^2\displaystyle\int_{f_0-B/2}^{f_0+B/2}df}=\frac{c^2\displaystyle\int_{-B/2}^{B/2}f^2df}{c^2B}=\frac{B^2}{12}\tag{3}\\\\$$
which equals your original (wrong) result for the low pass case. This is no surprise because a low pass signal with bandwidth $B/2$ becomes a band pass signal with bandwidth $B$ when modulated up to center frequency $f_0$.
