Front/back is actually very hard too, at least for stationary simulation without head tracking.
The main reason is simple: Left/Right is done by looking at the differences between the two ear signals, i.e. interaural level differences and interaural time differences. If a source is located to the left, the sound will arrive earlier on the left ear and will be louder (depending on frequency).
In the median plane (front, up, back down), the ear signals are almost identical. The only information available are pinna cues, shoulder reflection, interaural changes due to micro rotations of the head, small asymmetries, contextual information, etc. . That's a lot less robust so the localization acuity in general is a lot worse. It's also harder to model since it depends a lot on the individual human and one-size-fits-all doesn't work well.