How to detect rectangular images inserted into another image?

Question

It's just for a simple binary classification, so it's not needed to find the location of the inserted image or the content. I'm only interested in checking whether something like that is present or not.

Positive examples:

Negative examples:

The inserted image rectangles are always upright, i.e., they are not rotated by non-90-degree steps. They cover at least 1 % of the image area, i.e., their edges are at least 10 % of the image dimensions.

The images have undergone lossy compression to an unknown degree.

---

I've tried looking for straight horizontal and vertical edges using Scharr filters, Canny edge detection, and Hough lines, but the results are not very reliable. Any creative ideas would be appreciated. :)

---

My current attempt looks as follows:

import numpy as np
from cv2 import cv2


def get_value(image: np.ndarray) -> float:
    assert len(image.shape) and image.shape[2] == 3, "please provide a color image"

    scharr_scale = 1 / 16

    # Detect vertical and horizontal edges on all three channels (BGR) separately.
    scharr_x_abs = np.abs(cv2.Scharr(image, cv2.CV_32F, dx=1, dy=0, scale=scharr_scale))
    scharr_y_abs = np.abs(cv2.Scharr(image, cv2.CV_32F, dx=0, dy=1, scale=scharr_scale))

    # Reduce edge images to one channel.
    scharr_x_abs_gray = cv2.cvtColor(scharr_x_abs, cv2.COLOR_BGR2GRAY)
    scharr_y_abs_gray = cv2.cvtColor(scharr_y_abs, cv2.COLOR_BGR2GRAY)

    # Erode the edge images perpendicular to the filter direction to remove noise.
    scharr_x_abs_eroded = cv2.erode(scharr_x_abs_gray, np.ones((5, 1)))
    scharr_y_abs_eroded = cv2.erode(scharr_y_abs_gray, np.ones((5, 1)))

    # Get the average edge value per column/row part.
    scharr_x_row_abs = cv2.resize(scharr_x_abs_eroded, (scharr_x_abs_eroded.shape[1], 8), interpolation=cv2.INTER_AREA)
    scharr_y_col_abs = cv2.resize(scharr_y_abs_eroded, (8, scharr_x_abs_eroded.shape[0]), interpolation=cv2.INTER_AREA)

    # Build second derivative, but perpendicular to the original one,
    # to detect abrupt starts and ends of edges.
    scharr_x_row_abs_scharred_again = np.abs(cv2.Scharr(scharr_x_row_abs, cv2.CV_32F, dx=1, dy=0, scale=scharr_scale))
    scharr_y_col_abs_scharred_again = np.abs(cv2.Scharr(scharr_y_col_abs, cv2.CV_32F, dx=0, dy=1, scale=scharr_scale))

    # The maximum abrupt start/end of edges are the final result.
    scharr_x_row_max = np.max(scharr_x_row_abs_scharred_again)
    scharr_y_col_max = np.max(scharr_y_col_abs_scharred_again)
    result = float(np.maximum(scharr_x_row_max, scharr_y_col_max))

    return result

Output for the provides examples images:

• 100.99776458740234: A high value for this positive example. That's good.
• 37.71356201171875: A low value for this positive example. That's not good.
• 56.305137634277344: A not-too-high value for this positive example. That's ok.
• 27.330982208251953: A not-too-high value for this positive example. That's ok.

So there is no threshold that would divide the two classes correctly.

Answer 1

In the two positive examples, the rectangular box is "likely" to be noticed visually, notably by the frame, but also the difference of the content inside and outside the box, especially in the color diversity.

However, the compression (if lossy, JPEG_like) may induce local fake edges (quantization of the DC and higher frequency), local noise and fake colors along $8\times 8$ squares.

Hence, I would suggest a local/global algorithms in three steps.

First, estimation: a crude segmentation with closed contours. For instance with geodesic active regions or level sets. The principle is as follows: from a starting contour, the border evolve until it meet a region that either has a gradient (edge) and/or the statistics of the regions on both sides are different enough. Generally they are performered on colored gradient-tensors. Although the contour has some regularity, it is allowed to split and separate into several region. I think the is doable by maximizing the color contrast and allowing sharp edge

Second: model matching. You get a set of connected contours, which can be fitted by Cartesian rectangular boxes. The latter can be probed with your size constraints. One question remains: how to select only one box?

Third: decision/machine learning. Given a set of images, and a training ensemble of negatives and positives examples, you can train a logistic classifier on features related to the inner/outer boxes.

If you cannot learn examples, a distance between the feature-vector of the inner box to the feature-vector of the outer box could provide a quantitative metric related to some probability of having a rectangular image inserted.