# Detect non rectangular bounding box for objects

I have a short video where a TV is shown. I need to detect the TV there (let's assume it's all pure black or it's covered with a green screen), and overlay a video on top of it on the same area. But the problem is when the angle is not straight, it is oriented and the bounding boz is not a rectangle. So I guess I need to detect the angle and its scale to overlay the content correctly and nicely.

Is there any ways to do it with AI or computer vision? We can detect the TV with object detection, but I don't know how to compute the tilt and angle for best fit when it's not a retangle.

• Can you find the pixel coordinates of the four corners? – Cedron Dawg Aug 20 '20 at 1:46
• no. I don't know how to do that. Do you have a solution in mind? – Tina J Aug 20 '20 at 3:17
• That's actually the tougher part. Yes, I just started a company to do this kind of work. With having the four corners, defining a mapping and projecting an image is not that difficult. I developed my own chessboard calibration program (better than OpenCVs) where I can determine the edges to within about 1/100 or better of a pixel. This doesn't have as many parallel lines, but similar result should be achievable. Is this a commercial project or a personal one? – Cedron Dawg Aug 20 '20 at 3:27
• I see. I wish I could know more. It's just my personal experiment. What is that chessboard calibration? And u only detect rectangles? – Tina J Aug 20 '20 at 8:25
• Send me an email, cedron at exede dot net and I can send you a pic, and a little more information on algorithm, and an alternative you might use. You don't need that level of precision for this. There is a slick trick for mapping the pixels of your source onto the oblique rectangle on your screen (I'll post in a while), but for the easiest projection you want the inverse of that. – Cedron Dawg Aug 20 '20 at 12:00

Let's label your corners like this:

1-------2
|       |
3-------4


For your destination. Let, $$P_1, P_2, P_3, \text{ and } P_4$$ be vectors representing the corners in pixel coordinates.

For your source, Let $$H$$ and $$W$$ represent the height and width, and $$(x,y)$$ be a point in that rectangle you want to project.

You can calculate your coefficients like this:

\begin{aligned} C_1 &= \left(1 - \frac{x}{W} \right)\left( 1 - \frac{y}{H} \right) \\ C_2 &= \frac{x}{W} \left( 1 - \frac{y}{H} \right) \\ C_3 &= \left(1 - \frac{x}{W} \right)\frac{y}{H} \\ C_4 &= \frac{x}{W} \cdot \frac{y}{H} \end{aligned} Note: $$C_1 + C_2 + C_3 + C_4 = 1$$

Then the projection on the screen is at:

$$P = C_1 \cdot P_1 + C_2 \cdot P_2 + C_3 \cdot P_3 + C_4 \cdot P_4$$

This is a first order approximation that should be adequate for your needs. Especially if the TV image is a relatively small portion of the screen.

To locate the corners, a rough approach is to scan horizontally looking for black streaks. Match them row by row. The endpoints will define your TV edges. Use linear regression to fit a line. Next scan vertically in the same manner and do the same. Calculate the intersections as your corners. They need not be integer, and will be more accurate if you leave them fractional.

This is the easy way. Depending upon what you are projecting will likely work fine. The more proper way is to find the inverse mapping, then for every pixel on the screen, find the location in your source rectangle and do a weighted average there using the same splitting formula there to find the weights of the corners.

• Do we make an assumption that there is no other black area on the frame? – Tina J Aug 20 '20 at 15:44