3D model from high speed videos?

Question

Is there a way to convert or somewhat replicate RADAR data from a set of given high speed videos taken from multiple angles.

The problem is quite ill defined but the task I have in hand is to reproduce RADAR data for a set of videos of soccer shootouts. I also have a corresponding file with the coordinates of the ball, speed of it but not quite sure how that will come into play with this.

Answer 1

You'd basically need to do two things:

1. find a 3D-model of your environment, and track the ball in that. This model needs to have knowledge of the materials involved. Since a video gives you no information on the involved object's radar reflectivity, scattering coefficients and other RF parameters, that's something that you'll have to enhance your model with – manually.
2. Simulate the environment (e.g. via ray-tracing/volumetric RF FEM) as RF environment, and in that simulation emit your radar signal and observe the echo.

Honestly, I don't think this is overly viable ask "task I have", unless "I" is very much more experienced in RF simulations than I am, or "I" is a medium to large team of researchers.

Answer 2

I do not have expertise or experience in image processing but find this interesting, so I created the graphic below to assist in finding the solution or establishing why this can't be done.

If we knew the precise location and orientation of the cameras in 3d space, as well as their focal length and lens distortions (which could be calibrated against reference conditions)- and had multiple images or video, could we not extract position and velocity information on a target moving from location T1 to location T2? What else would need to be known and what are other limitations?

I would also think that if the remaining part of the image is not moving, we could perhaps further eliminate some of the constraints in what must be known in order to resolve.

Answer 3

Is there a way to convert or somewhat replicate RADAR data from a set of given high speed videos taken from multiple angles.

Yes. But it is not going to be a walk in the park.

The problem is quite ill defined but the task I have in hand is to reproduce RADAR data for a set of videos of soccer shootouts. I also have a corresponding file with the coordinates of the ball, speed of it but not quite sure how that will come into play with this.

I am not sure what this data looks like or how it is produced but let's say that our output is $x,y,z,v$ of the ball.

The fundamental concept here is "Stereovision". Which is trivial with two cameras whose positions and orientation is known. If you were to put two web cams $d$ meters appart, with a known focal length $f$, then you would be solving for the "height" of a triangle out of which you know the length of its base ($d$) and the two angles adjacent to the base. I hope you can see the triangle here, its base is the two cameras and its apex is the ball when it can be viewed by two cameras. So in a very simple setting, if you had a GUI that gave you a stream from each camera, you could freeze frame it, click on an object on one image ($x_1, y_1$), click on the same object on the other image ($x_2, y_2$) and work out their distance in physical space (The height of the triangle).

To avoid the "click here, click there", you would have to match where the user clicked on one camera with where a similar patch of pixels is on the image from the other camera. For the general case of matching between two images, this has been done to exhaustion. For the specific case of matching exactly what the user clicked on, you could use something like normalised correlation (again, this has been done to exahustion).

OK, so, to do stereo-vision we need $(x_1, y_1), (x_2, y_2)$ in which case the next problem to solve is "Find the ball" within the video stream.

Again, this has been done to exhaustion. You could for example train YOLO for your specific problem and then let it find bounding boxes that are likely to contain balls. The centroid of the identified areas between two independent video stream frames (from the same moment) give you the $(x_1, y_1), (x_2, y_2)$.

Obviously, since you have a video, you would track the "height" of the triangle between frames $n, n+1$ and through the coordinates of the ball and the video FPS you would work out the ball's velocity.

So, the "easy" option is to shoot a game from two fixed cameras that are looking straight ahead and are at a known distance from each other.

The horrible details are:

1. You need to synchronise the video streams between all cameras so that the $n_{th}$ frame from the $k_th$ camera refers to the exact same moment in time. Otherwise, your inferred positions (and velocities later on) will be in error.

2. The fact that you have more than one cameras does not mean that the ball is simultaneously visible by all of them, so this inserts an intermediate stage where you decide which subset of cameras is the best to infer the ball's position based on the output of the model that finds the ball for you. For example, the ball might only be visible from two out of three cameras, you would then have to infer position based on the geometry of those two specific cameras.

3. Cameras do not come in pairs looking straight ahead. Each of the fixed cameras following a match is at a specific position but they might be tracking the ball at a different angle. This means that before you use the $x_1, y_1$ (for example) you would have to adjust them for the specific viewpoint of the camera. This is....a bit difficult because you need to know how it is rotated (in space) and you would probably need a hardware sensor on the camera for this (there might be cameras that return this information readily, I think that open camera (for android) does this so it should be available in professional cameras too.). Obviously, if you don't have the rotation of the camera you are going to have to infer it from the video (or at least a subset of the frames). For example: find the goal post, infer angle from "skewness" of the rectangle, ignore "silly" solutions knowing the position of the camera. But the goal post might not be visible at all times.

4. Along with rotation, a camera that is tracking a ball also zooms in to the ball which means that you also need to be tracking the individual $f$ from each camera because they participate in working out the "height" of the triangle. You could use the "scale" of the ball but you would have to have that calibrated for the camera you are using.

Once these problems are straightened out, you could then use the extracted data for the position of the ball for a game with the ground-truth data (from the radar) in the validation step.

Hope this helps.