I'm looking for a formula to calculate the bit error rate of a wifi communication 802.11a/b/g according to the distance between transmitter and receiver.
There is no one-line "formula", if that is what you are looking for. Traditionally, BER is captured at link-level, meaning it considers the effect of:
- Modulation (BPSK, QPSK, etc.)
- Channel coding rate
- Multipath fading and channel conditions (eg. AWGN, Rayleigh fading, etc.)
However, once you talk about distance between TX-RX, we are talking about propagation effects, shadowing, which are part of system-level simulations.
Moereover, your question is ambiguous: you simply state 802.11a/b/g without mentioning whether you are talking of OFDM or DSSS, what channel bandwidth, etc.
Depending on the scenario, there are lots of IEEE papers, ex: BER simulation for WLAN networks in realistic environment