Detecting bullet holes using Python with camera or sensors

I am trying to build a system to detect bullet holes in paper. I've read many StackOverflow threads but not a single one satisfied me. Current methods, which I am using, are good but not perfect and not fully reliable. It often misses couple of shots or has problems with calibration when live testing.

System requirements:

• Realtime detection,
• Has very low input lag between bullet making hole in paper and detection,
• minimal screen size - 2.5m x 1.5m,
• works with .22, 9x19, 5.56x45, 7.62x39 etc. ammunition,
• returns X and Y coordinates of detected bullet hit.

What I am currently using:

• Arducam IMX477 12,3MPx HQ + usb 2.0 interface to simulate webcam,
• Camera lens - which is terrible :P but i needed zoom function, visible light filter on camera and IR lights - to be independent from enviroment lights.
• Python detection:
• Image subtraction - monitoring changes,
• Thresholding an image from camera - to eliminate noise,
• Detection - OpenCV.findContours or analysis of numpy array in search of high values in some region of image.

Camera is located 4.5-5m from screen. I'm detecting black areas that appear on following frames by watching for changes between these frames and finding maximum if some big spike (2 times greater than noise) is detected.

Here is some code example and nice plots that I've made:

import cv2
import time

video_stream = cv2.VideoCapture('C:\Projekty\HoleDetection\data\IRNEW\WIN_20220204_14_26_23_Pro_Trim.mp4')
prev_frame = None
i = 0
start = float(time.time())
while video_stream.isOpened():
if not ret:
break
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

if prev_frame is not None:
subtracted_frame = cv2.subtract(prev_frame, frame)
# cv2.imshow('subtr', cv2.resize(subtracted_frame, (1280, 720)))

ax_0_max = subtracted_frame.max(axis=0)
ax_1_max = subtracted_frame.max(axis=1)
curr_max = ax_1_max.max()

if prev_max * 2 < curr_max:
print('Detected shot')
x = ax_0_max.argmax()
y = ax_1_max.argmax()
print(f'X: {x}, Y: {y}')
# keypoints.append(cv2.KeyPoint(float(x), float(y), 10))
prev_max = curr_max
else:
prev_max = frame.max()
prev_frame = frame
# frame = cv2.drawKeypoints(frame, keypoints, np.array([]), (0, 0, 255), cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
# cv2.imshow('im', cv2.resize(frame, (1280, 720)))
i += 1
if cv2.waitKey(1) == ord('q'):
break
end = float(time.time())
cv2.destroyAllWindows()

print(f'Time {end-start} for {i+1} frames. FPS: {(i+1)/(end-start)}')


Whole movie analysis:

Keyframe where first bullet hole appeared:

This is how it looks like from camera perspective. Little bit blurry and fish lens effect. It's not ideal. It works almost perfect (or maybe just good) with 9x19 ammunition, but has problems with smaller calliber (.22, .223Rem, 5.56x45) - changes are less noticeable because of smaller holes.

Notice one unfortunate thing - I can't put anything behind the screen because it's shelled with live ammunition.

I would like to hear from you how to improve this system. Maybe use some kind of sensors that are capable of detecting bullet holes (if there is such a thing) and more reliable than current methods or maybe there is some better python method available

I can't really comment on the machine vision part, other than any question that asks "How do I do <some signal processing task> in <some language>" is fairly naive. The way you do any signal processing task in any language is to first learn how to do the signal processing in math. Full stop. Then go and implement that in the language of your choice.

All the major computing languages share all the major math, signal processing, and machine vision libraries. And for the ones that don't, there's usually a way to put a thin wrapper around the library of your choice and use it anyway. So if you want to do it in Python vs. C++ or Rust or Fortran or whatever -- that's just a detail. The important part is how you do it in the first place.

For your problem at hand -- I think you just need more contrast, and possibly more pixels. Work on your lens, your camera, and your lighting to make the bullet holes more distinctive. If those are 9mm holes and you want to go even smaller, then you need to have a high pixel count camera, and you need optics on it to match. You probably want your bullet holes to be at least five pixels in diameter, and more is always better.

You may want to rethink your "no backlight" rule. E.g., if you set the camera up to view the front of the target from the left, out of range, then couldn't you set up lights behind the target and to the right, so they shine right into the camera but are out of the path of the bullets?

• GIGO is always a problem with image processing problems. Algorithms can do much, but if you don't start with decent input, it's hard to make it decent via algorithms alone.
– Peter K.
Mar 25 at 17:57
• And it's hard to just look at something and decide if it's good enough. Our ancestors were experimenting with neural-net based vision 500 million years ago, but we've only been migrating that to machine vision for the last 50 or 60 years. Thinking that some machine vision task will work OK because we can see it often leads to disappointment. Mar 25 at 19:29
• Yes. A guy I know who researches (deeply) into learning systems calls NNs "neutral notworks". :-)
– Peter K.
Mar 25 at 19:38

You will get better results using an infrared camera and lock-in thermography. The infrared reflection of bullets is much more different than the possibilities with 'normal' light. So you will get more details.