In general, you can apply a filter with some time constant T which will be giving you a more reliable RSSI value but even in this case, the RSSI will only be an accurate indication of distance under certain assumptions.
There are two types of filters you could use here, a moving average low pass filter which would effectively produce a smoothed out version of the RSSI waveform or a non-linear modal-mean or median filter. An example of an easily implemented moving average Finite Impulse Response (FIR) filter would be h=ones(N)./N where N is the order of the filter that is directly proportional to the Group Delay (or time constant) of the filter (for instance a [0.25,0.25,0.25,0.25] N=4 filter). Applied over a sliding window of N samples, this filter would return an average, slow changing, estimate of the RSSI curve which you could then associate with distance. Similarly, the non-linear filter is applied over a sliding window of size N but in this case you just estimate the modal mean (i.e. the most frequently occurring RSSI value of the N available in the sample) or the median value. The median filter would tend to remove the spikes and could possibly be a better alternative for this application.
The relationship between RSSI and distance would be established through the Free Space Path Loss but the main assumptions here would be that a) there is a direct line-of-sight between transmitter and receiver and b) the RSSI is accurately estimated by the device. In other words, if you start moving the device in space, it is not necessary that the distance estimation will remain accurate. You might also want to check this link as well.
Hope this helps.