There is already a succesful question How did they color old black and white movies? on this site. Very general and correspondingly in general answered. Let me please narrow it down to just the photograpy (i.e. no time dimension) and more specified starting point:
What is the algorithm to color the greyscale HD picture given the colors at various points on greyscale original?
Note: Yes, I am aware that the original colors can't be restored any more.
... by using vector masks to indicate regions of uniform color on a few keyframes, and interpolating between them. Then a color transform is applied to each mask.
And then the last paragraph on automating the process. $\endgroup$