I believe the answer (bilateral is good) can be balanced by the luminance quantization (from 1-bit black and white or true color), and the fact that your image is very small, so standard statistics don't apply here.
Indeed, a mean over $4$ pixels is generally not an integer, and will be rounded at the end. If you stick to $2\times2$ blocks, I would suggest to take the median of the four values: get rid of the
min and the
max of the four, and take the "nearest" integer (
floor depending on whether you want a darker or brighter icon), and the
max of the mean of the two remaining pixels. I expect this to limit "intermediate luminance borders" that may not be useful at this size, and avoids grays on low-quantization images. I would treat the corners by "duplicating" the corner value (to limit the chances of an ugly color change at the borders): you will get $5$ pixel values, and then the standard median will be an integer.
More involved processing may resort to averaging $4\times 4$ patches, with a lower weight for the 10 surrounding pixels.
Of course, the effectiveness depends on the morphology of yoour image: is it flat, smoothly varying or stochastic?
Honestly, I suspect this proposal won't change dramatically the result on standard displays.