Interesting experiment.
What I believe would be needed are repeated measurements in both known good and known bad conditions to establish the metrics of the random variables involved from which we could establish an optimum decision approach.
The variability needs to be established from both other factors in the physical units (tolerance of bolt locations, torque etc) and in the measurement set-up itself (variation of placement in test stand, repeatability of force and strike location of instrumented hammer). Finally we need to establish how stationary these statistics are, are the results repeatable with time, and is the process ergodic where repeated measurements can be used to establish the needed estimates of such statistic.
This is identical to knowing what the noise floor of your measurement system is, and what the signal to noise (SNR) of your unit under test is, and if the noise floor is too high compared to the SNR, the duration of time for which stationarity can be assumed can be used to establish SNR improvement through repeated measurements and averaging.
Once the statistical results are estimated, further approaches can be established to make the classic trade of probability of false alarm versus probability of detection, minimizing the former while maximizing the latter.
To proceed with this approach the next step is to establish those statistics, and as typically done, get as many measurements possible for both cases of known good units, and known bad units, while monitoring validity of stationarity (statistically are the results consistent when reviewing from test to test, so a plot similar to the lower one, but each column represents a different test, and from that we can see if results are drifting; I suspect we can assume it is stationary but without seeing all aspects of the process it is something I still like to confirm from the results-- unlikely you would get enough test samples to really see this initially but can be something that the "factory" monitors over time).
Known good unit testing: This will establish the "noise floor" of the test as well as unit to unit variability and would be good to isolate both factors. For the test stand itself, use a known good unit, installed and repeat the measurement as many times as possible (I would start with at least 20 times, and then when the entire process is complete and clearer assess if higher accuracy of the derived statistics are even needed). This should be done with the unit and installed by the operator in each test (and in an actual production situation, the entire test would be tracked over time with feedback from some other pass/fail test, perhaps sample testing with a higher degree of inspection, to then establish how the underlying test metrics may have drifted such as over time or from operator to operator). From this, create the same plot of "frf results" and establish the statistics for the variability.
Then repeat the above using a known bad unit (at what point do you establish the bolt to be "loose"? I assume here there is a minimum torque requirement so the test should be done with a unit at this threshold condition) with the experiment otherwise exactly as done above repeated. Then the test should be done across multiple units. Finally the test should be repeated at above threshold condition, to show/ confirm that the results clearly move in the right direction for establishing a good/bad condition.
You could extend this to machine learning concepts to classify the loose bolt condition, but I am thinking much simpler than that with much smaller data sets— simple statistics to quantify the random variables that would otherwise cloud your good vs bad decision metric.