suppose if we send AM modulated message signal through space how the receiver will be knowing that we applied AM modulation basically how it will be knowing that which modulation is applied for our modulated signal
Communications systems are always designed under the assumption that both emitter and receiver know what "language" they will be speaking to each other.
AM modulation comms are standardized in frequency, channel width for example. So each receiver is materially designed to demodulate AM signals with the appropriate hardware.
This is a signal intelligence version of a randomness test (plus possibly a Turing test).
A receiver (of many simple modulation schemes) can "demodulate" any signal, but the result might just be noise.
Given a slice of spectrum, one can try analyzing it to estimate the likelihood that it is random or perhaps varied by some non-random process. Given that one finds some non-random modulation(s), one might try estimating whether that modulation was produced by a physical process or holds much more information than can be produced by any known non-intelligent mechanism, thus indicating a signal. But a smarter spy might try to make their modulation look like random noise rather than patterned or intelligent to try and fool you. Game On.