Detecting bullet holes using Python with camera or sensors

Question

I am trying to build a system to detect bullet holes in paper. I've read many StackOverflow threads but not a single one satisfied me. Current methods, which I am using, are good but not perfect and not fully reliable. It often misses couple of shots or has problems with calibration when live testing.

System requirements:

• Realtime detection,
• Has very low input lag between bullet making hole in paper and detection,
• minimal screen size - 2.5m x 1.5m,
• works with .22, 9x19, 5.56x45, 7.62x39 etc. ammunition,
• returns X and Y coordinates of detected bullet hit.

What I am currently using:

• Arducam IMX477 12,3MPx HQ + usb 2.0 interface to simulate webcam,
• Camera lens - which is terrible :P but i needed zoom function, visible light filter on camera and IR lights - to be independent from enviroment lights.
• Python detection:
  • Image subtraction - monitoring changes,
  • Thresholding an image from camera - to eliminate noise,
  • Detection - OpenCV.findContours or analysis of numpy array in search of high values in some region of image.

Camera is located 4.5-5m from screen. I'm detecting black areas that appear on following frames by watching for changes between these frames and finding maximum if some big spike (2 times greater than noise) is detected.

Here is some code example and nice plots that I've made:

import cv2
import time

video_stream = cv2.VideoCapture('C:\Projekty\HoleDetection\data\IRNEW\WIN_20220204_14_26_23_Pro_Trim.mp4')
prev_frame = None
i = 0
start = float(time.time())
while video_stream.isOpened():
    ret, frame = video_stream.read()
    if not ret:
        break
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    if prev_frame is not None:
        subtracted_frame = cv2.subtract(prev_frame, frame)
        # cv2.imshow('subtr', cv2.resize(subtracted_frame, (1280, 720)))

        ax_0_max = subtracted_frame.max(axis=0)
        ax_1_max = subtracted_frame.max(axis=1)
        curr_max = ax_1_max.max()

        if prev_max * 2 < curr_max:
            print('Detected shot')
            x = ax_0_max.argmax()
            y = ax_1_max.argmax()
            print(f'X: {x}, Y: {y}')
            # keypoints.append(cv2.KeyPoint(float(x), float(y), 10))
        prev_max = curr_max
    else:
        prev_max = frame.max()
    prev_frame = frame
    # frame = cv2.drawKeypoints(frame, keypoints, np.array([]), (0, 0, 255), cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
    # cv2.imshow('im', cv2.resize(frame, (1280, 720)))
    i += 1
    if cv2.waitKey(1) == ord('q'):
        break
end = float(time.time())
cv2.destroyAllWindows()

print(f'Time {end-start} for {i+1} frames. FPS: {(i+1)/(end-start)}')

Whole movie analysis:

Keyframe where first bullet hole appeared:

This is how it looks like from camera perspective. Little bit blurry and fish lens effect. It's not ideal. It works almost perfect (or maybe just good) with 9x19 ammunition, but has problems with smaller calliber (.22, .223Rem, 5.56x45) - changes are less noticeable because of smaller holes.

Notice one unfortunate thing - I can't put anything behind the screen because it's shelled with live ammunition.

I would like to hear from you how to improve this system. Maybe use some kind of sensors that are capable of detecting bullet holes (if there is such a thing) and more reliable than current methods or maybe there is some better python method available

Answer 1

I can't really comment on the machine vision part, other than any question that asks "How do I do <some signal processing task> in <some language>" is fairly naive. The way you do any signal processing task in any language is to first learn how to do the signal processing in math. Full stop. Then go and implement that in the language of your choice.

All the major computing languages share all the major math, signal processing, and machine vision libraries. And for the ones that don't, there's usually a way to put a thin wrapper around the library of your choice and use it anyway. So if you want to do it in Python vs. C++ or Rust or Fortran or whatever -- that's just a detail. The important part is how you do it in the first place.

For your problem at hand -- I think you just need more contrast, and possibly more pixels. Work on your lens, your camera, and your lighting to make the bullet holes more distinctive. If those are 9mm holes and you want to go even smaller, then you need to have a high pixel count camera, and you need optics on it to match. You probably want your bullet holes to be at least five pixels in diameter, and more is always better.

You may want to rethink your "no backlight" rule. E.g., if you set the camera up to view the front of the target from the left, out of range, then couldn't you set up lights behind the target and to the right, so they shine right into the camera but are out of the path of the bullets?

Answer 2

You will get better results using an infrared camera and lock-in thermography. The infrared reflection of bullets is much more different than the possibilities with 'normal' light. So you will get more details.

Answer 3

A couple of comments from a practical aspect, I can't comment on the electronics side - close grouping of shots will be an issue unless you an automatic paper roller feeder system which "closes up" the paper periodically. Other substrates will be an issue in the long term such as latex rubber anti splinter screens.

Placement of the camera needs to be considered - certain target systems do not have the space or orientation to get the camera in place. It will all depend on the training being undertaken.

Thermal systems have their issues with accuracy, again down to close grouping on rapid fire practices.

As a benchmark - gold standard accuracy is half a calibre i.e. approx. 4.5mm for a 9mm round. This is the best that that a conventional acoustic target will perform at.

You could consider a machine learning "light screen" in front of the shooting surface.