Calculating satellite position from images taken by the satellite of the earth surface

Question

I'm currently a little stuck with a problem, that sounds easier than it is (at least for me):

Let's say you have satellite images taken from LEO that show an approximately 1000 km wide area (the image axis of the camera is more or less perpendicular to the ground). There is no additional location data stored in the image, so no way of directly extracting the position the image was taken).

What I want to do is write a program (in python on a raspberry pi with coral usb) that can find the location the image was taken from by matching it to a map of earth. this should be done automatically (more or less in real time) for the purpose of calculating the orbit of the satellite taking the images.

I've no problem calculating the orbit, once I have location data (even if it's very noisy), using a technique based on an Extended Kalman Filter.

Matching a satellite image to a map of earth by just using the image data, on the other hand.... I honestly don't even know where to start.

I know this is an incredibly unspecific question and not related to a specific problem, but maybe someone could point me in the right direction...

Just to give you an idea how those unprocessed images from LEO look, I included a few reasonably good images taken over one orbit of earth.

I took those images with a NIR camera through a small port hole on the International Space Station last year. Resolution of the images I included have been only 640x480 (by mistake!), but image resolution should be around 4k.

These images have some artifacts in them due to the fact that they where taken through a thick glass window of the ISS - so there are some reflections happening there...

Answer 1

There are three approaches to this:

1. The ISS is in a specific orbit above the Earth. You took the images on a specific date and time. You can query exactly where the ISS was at that date and time and this will at least put you in the area where you are more likely to be able to recognise the features you were photographying.

2. Post the images to be recognised by human beings from around the globe in a crowd-sourcing "experiment".

3. One of the earliest misile navigation approaches was "Terrain Contour Matching (TERCOM)". TERCOM works by cross correlating the ground relief in flight (as sensed by various sensors) to a stored version of the ground relief and activates control surfaces to maximise that correlation.

   In your case, you have the in-flight images which you can start comparing to a stored version of the Earth's relief. A freely available dataset is the "Shuttle Radar Topography Mission".

   BUT, there is a "small" caveat: Your images show what near infra-red light "colour" the planet was reflecting back to your camera. Each pixel of the SRTM dataset contains the distance between the ground and the sensor in space. Therefore, your "best" results will be obtained by using images 1 and 4 that show some ground features (rivers / lakes, etc). More on this later, but keep this fundamental difference in mind for the moment.

   The SRTM comes in two spatial accuracies 90 meters (SRTM90) and 30 meters (SRTM30). Here is a convenient downloader for each tile.

   Each tile in the SRTM dataset is a GeoTiff image that describes a specific "sector" on the Earth's surface.

   So, to find "where on Earth" was an in-flight image containing stark ground features taken, you would have to solve an image registration problem between the in-flight image and each relief tile from across the Earth. Specifically, a "Rigid Body Registration" would return the "center" of the in-flight image and the scaling and rotation it would have to undergo to match the corresponding ground feature. Also, this would work best if you were to convert your in-flight images to grayscale. Your chances of finding an exact match are reduced if the ground features are covered by clouds and if your camera lens / observation apperture adds non-linear distortions to the images (which, they do, usually though at the edges of the frame. Same goes for the observation window)

Now, number 3 is a "last ditch" solution. And this is because we are talking about a large amount of processing before you can have an idea of where you can find the...."global minimum" (in this case, almost literally the "global" minimum).

A parameter that would constrain your search space would be to look only in regions that contain rivers. You might find a segmentation layer for the SRTM but you might be able to get big enough rivers using relatively simple heuristics.

Number 3 would be assisted by findings of 1 and 2. So, your crowd-sourcing could return the approximate area most of your respondents think these pictures are from (if not, the exact location) which you could use to constrain the number of tiles you search over.

Hope this helps.