# 3D model from high speed videos?

Is there a way to convert or somewhat replicate RADAR data from a set of given high speed videos taken from multiple angles.

The problem is quite ill defined but the task I have in hand is to reproduce RADAR data for a set of videos of soccer shootouts. I also have a corresponding file with the coordinates of the ball, speed of it but not quite sure how that will come into play with this.

• This post may help: photo.stackexchange.com/questions/71923/… – Dan Boschen Jan 7 at 12:58
• And this one may be extended to your application: pyimagesearch.com/2015/01/19/… I assume your question really is how to extract position and velocity information of a moving target from multiple videos-- if so you might improve your title by removing RADAR to eliminate the need for RF reflectivity, scattering etc in the answers, which isn't related to video images, I assume you can extract velocity and position assuming you have info about your lens, focal distance etc -- – Dan Boschen Jan 7 at 14:00
• It would be more accurate to substitute "3D model" or "3D world model" for "RADAR data". To me, at least, "radar" specifically means "radio" (as "lidar" is the light analog of radar). – TimWescott Jan 7 at 15:54
• and to add LIDAR is with lasers specifically, so replacing with LIDAR also would not be applicable-- Tim's suggestion of "3D model" is excellent- Unless what you really want is just velocity and or position – Dan Boschen Jan 7 at 17:50
• 3d model seems like the best fit with the data I have in hand. Any recommendations on where should I start with that? – sspatole Jan 7 at 18:01

You'd basically need to do two things:

1. find a 3D-model of your environment, and track the ball in that. This model needs to have knowledge of the materials involved. Since a video gives you no information on the involved object's radar reflectivity, scattering coefficients and other RF parameters, that's something that you'll have to enhance your model with – manually.
2. Simulate the environment (e.g. via ray-tracing/volumetric RF FEM) as RF environment, and in that simulation emit your radar signal and observe the echo.

Honestly, I don't think this is overly viable ask "task I have", unless "I" is very much more experienced in RF simulations than I am, or "I" is a medium to large team of researchers.

• Well thanks, unfortunately "I" is just me at the moment and I don't have many resources available at the moment. All I have is a few videos and that's about it. Is there an easier or more feasible alternative to go around this? – sspatole Jan 7 at 12:52
• He is using photographic images, so RF, reflectivity, scattering etc doesn't really come into play here, does it? – Dan Boschen Jan 7 at 12:57
• @DanBoschen yeah, right. – sspatole Jan 7 at 12:59
• @DanBoschen: If the idea is to construct a dataset that looks like it came from a radar, then you will need some way to estimate RF reflectivity, scattering, etc. from the video data so that your simulated radar data is somewhat realistic. – JRE Jan 7 at 13:09
• @JRE the idea is to reconstruct a dataset to appear as if it came from RADAR. Are there any ways to estimate RF reflectivity from video data? – sspatole Jan 7 at 13:12

I do not have expertise or experience in image processing but find this interesting, so I created the graphic below to assist in finding the solution or establishing why this can't be done.

If we knew the precise location and orientation of the cameras in 3d space, as well as their focal length and lens distortions (which could be calibrated against reference conditions)- and had multiple images or video, could we not extract position and velocity information on a target moving from location T1 to location T2? What else would need to be known and what are other limitations?

I would also think that if the remaining part of the image is not moving, we could perhaps further eliminate some of the constraints in what must be known in order to resolve.

Is there a way to convert or somewhat replicate RADAR data from a set of given high speed videos taken from multiple angles.

Yes. But it is not going to be a walk in the park.

The problem is quite ill defined but the task I have in hand is to reproduce RADAR data for a set of videos of soccer shootouts. I also have a corresponding file with the coordinates of the ball, speed of it but not quite sure how that will come into play with this.

I am not sure what this data looks like or how it is produced but let's say that our output is $$x,y,z,v$$ of the ball.

The fundamental concept here is "Stereovision". Which is trivial with two cameras whose positions and orientation is known. If you were to put two web cams $$d$$ meters appart, with a known focal length $$f$$, then you would be solving for the "height" of a triangle out of which you know the length of its base ($$d$$) and the two angles adjacent to the base. I hope you can see the triangle here, its base is the two cameras and its apex is the ball when it can be viewed by two cameras. So in a very simple setting, if you had a GUI that gave you a stream from each camera, you could freeze frame it, click on an object on one image ($$x_1, y_1$$), click on the same object on the other image ($$x_2, y_2$$) and work out their distance in physical space (The height of the triangle).

To avoid the "click here, click there", you would have to match where the user clicked on one camera with where a similar patch of pixels is on the image from the other camera. For the general case of matching between two images, this has been done to exhaustion. For the specific case of matching exactly what the user clicked on, you could use something like normalised correlation (again, this has been done to exahustion).

OK, so, to do stereo-vision we need $$(x_1, y_1), (x_2, y_2)$$ in which case the next problem to solve is "Find the ball" within the video stream.

Again, this has been done to exhaustion. You could for example train YOLO for your specific problem and then let it find bounding boxes that are likely to contain balls. The centroid of the identified areas between two independent video stream frames (from the same moment) give you the $$(x_1, y_1), (x_2, y_2)$$.

Obviously, since you have a video, you would track the "height" of the triangle between frames $$n, n+1$$ and through the coordinates of the ball and the video FPS you would work out the ball's velocity.

So, the "easy" option is to shoot a game from two fixed cameras that are looking straight ahead and are at a known distance from each other.

The horrible details are:

1. You need to synchronise the video streams between all cameras so that the $$n_{th}$$ frame from the $$k_th$$ camera refers to the exact same moment in time. Otherwise, your inferred positions (and velocities later on) will be in error.

2. The fact that you have more than one cameras does not mean that the ball is simultaneously visible by all of them, so this inserts an intermediate stage where you decide which subset of cameras is the best to infer the ball's position based on the output of the model that finds the ball for you. For example, the ball might only be visible from two out of three cameras, you would then have to infer position based on the geometry of those two specific cameras.

3. Cameras do not come in pairs looking straight ahead. Each of the fixed cameras following a match is at a specific position but they might be tracking the ball at a different angle. This means that before you use the $$x_1, y_1$$ (for example) you would have to adjust them for the specific viewpoint of the camera. This is....a bit difficult because you need to know how it is rotated (in space) and you would probably need a hardware sensor on the camera for this (there might be cameras that return this information readily, I think that open camera (for android) does this so it should be available in professional cameras too.). Obviously, if you don't have the rotation of the camera you are going to have to infer it from the video (or at least a subset of the frames). For example: find the goal post, infer angle from "skewness" of the rectangle, ignore "silly" solutions knowing the position of the camera. But the goal post might not be visible at all times.

4. Along with rotation, a camera that is tracking a ball also zooms in to the ball which means that you also need to be tracking the individual $$f$$ from each camera because they participate in working out the "height" of the triangle. You could use the "scale" of the ball but you would have to have that calibrated for the camera you are using.

Once these problems are straightened out, you could then use the extracted data for the position of the ball for a game with the ground-truth data (from the radar) in the validation step.

Hope this helps.