I am using images and corresponding lidar values for depth estimation CNN. I have RGB images. Corresponding to each RGB images, there is another image representing a subsets of pixels and distances to these pixels.
RGB image is of size 1920*1080 pixels. Corresponding distance image has approximately 500000 points. Each of these points are in the following format [x-coordinate, y-coordinate, distance-to-point, Red,Green, Blue].
A sample distance image is shown. The x,y coordinates and RGB values are shown in the image. The distance information is not encoded. Points with no distance values are set as black 
My problem is that, when i superimpose the original RGB image and this subset of points, there is a shift in these points. The X,Y coordinates are not matching the original image.
I think that this is because of lidar calibration issues. But I dont have any lidar information available.
The shift consists of Rotational and translation shifts.
How can I correct this shift and align x,y points correctly?
Any tool or any mathematical model or any other suggestion is welcome.