The mathematical procedure asked, is right and you may be missing something.
Let's assume a room with concrete walls and floor.
The speed of sound in air is around 300m/s.
The speed of sound in concrete is around 3400m/s and concrete supports slower sheer waves are about 1900m/s and there are also Rayleigh waves.
In a big room and at frequencies where the coupling from air to concrete is negligible, you can usually expect that the direct acoustic path with be the shortest.
There are also cases where you get coupling (like a speaker sitting on the floor) and the concrete path (P-wave) will arrive first through the floor to your sensor supported on a metal stand. Let's say you suspend your sensor on a string attached the ceiling. Even if the sensor is in the exact same place, the impulse response will not be the same. In other words, your sensor setup is part of the impulse response. Now fill the room with people and the impulse response is changed again.
There is also the practical matter that an ideal impulse response has an infinite bandwidth so necessarily, you need to restrict the frequencies to some band, and that depends on your actual problem.
None of this says that the simple direct line-of-sight air path is wrong, but I wouldn't count on repeatable results. It's also easy to overthink the physics.