You can try doing edge detection on individual color domains and then merge them, using your method of choice for edge detection.

Compared to edge detection directly on the color image, it might produce better results.