It's just for a simple binary classification, so it's not needed to find the location of the inserted image or the content. I'm only interested in checking whether something like that is present or not.
Positive examples:
Negative examples:
The inserted image rectangles are always upright, i.e., they are not rotated by non-90-degree steps. They cover at least 1 % of the image area, i.e., their edges are at least 10 % of the image dimensions.
The images have undergone lossy compression to an unknown degree.
I've tried looking for straight horizontal and vertical edges using Scharr filters, Canny edge detection, and Hough lines, but the results are not very reliable. Any creative ideas would be appreciated. :)
My current attempt looks as follows:
import numpy as np
from cv2 import cv2
def get_value(image: np.ndarray) -> float:
assert len(image.shape) and image.shape[2] == 3, "please provide a color image"
scharr_scale = 1 / 16
# Detect vertical and horizontal edges on all three channels (BGR) separately.
scharr_x_abs = np.abs(cv2.Scharr(image, cv2.CV_32F, dx=1, dy=0, scale=scharr_scale))
scharr_y_abs = np.abs(cv2.Scharr(image, cv2.CV_32F, dx=0, dy=1, scale=scharr_scale))
# Reduce edge images to one channel.
scharr_x_abs_gray = cv2.cvtColor(scharr_x_abs, cv2.COLOR_BGR2GRAY)
scharr_y_abs_gray = cv2.cvtColor(scharr_y_abs, cv2.COLOR_BGR2GRAY)
# Erode the edge images perpendicular to the filter direction to remove noise.
scharr_x_abs_eroded = cv2.erode(scharr_x_abs_gray, np.ones((5, 1)))
scharr_y_abs_eroded = cv2.erode(scharr_y_abs_gray, np.ones((5, 1)))
# Get the average edge value per column/row part.
scharr_x_row_abs = cv2.resize(scharr_x_abs_eroded, (scharr_x_abs_eroded.shape[1], 8), interpolation=cv2.INTER_AREA)
scharr_y_col_abs = cv2.resize(scharr_y_abs_eroded, (8, scharr_x_abs_eroded.shape[0]), interpolation=cv2.INTER_AREA)
# Build second derivative, but perpendicular to the original one,
# to detect abrupt starts and ends of edges.
scharr_x_row_abs_scharred_again = np.abs(cv2.Scharr(scharr_x_row_abs, cv2.CV_32F, dx=1, dy=0, scale=scharr_scale))
scharr_y_col_abs_scharred_again = np.abs(cv2.Scharr(scharr_y_col_abs, cv2.CV_32F, dx=0, dy=1, scale=scharr_scale))
# The maximum abrupt start/end of edges are the final result.
scharr_x_row_max = np.max(scharr_x_row_abs_scharred_again)
scharr_y_col_max = np.max(scharr_y_col_abs_scharred_again)
result = float(np.maximum(scharr_x_row_max, scharr_y_col_max))
return result
Output for the provides examples images:
100.99776458740234
: A high value for this positive example. That's good.37.71356201171875
: A low value for this positive example. That's not good.56.305137634277344
: A not-too-high value for this positive example. That's ok.27.330982208251953
: A not-too-high value for this positive example. That's ok.
So there is no threshold that would divide the two classes correctly.