My goal is to make an approach to recognize modulation type of unknown signals.
Cool! Automatic modulation classification is still an exciting topic, after it was (one of) the foundations from which in the 1980s Software Defined Radio arose, e.g. [1].
Do you think that problem is unsolvable and this is waste of time?
Absolutely not! There's quite a few people and companies doing this, classically, commercially selling tools to automatically classify and demodulate signals [3], or SDKs with which you can build your own Machine Learning-based classificators [4].
Problem is that I don't know exact sampling rate at which desired signal was generated at the transmitter.
you don't really care about the sampling rate (can't tell that from the signal, ever; the transmitter contain a, say, 5/7-rate resampler that doesn't change the signal at all), but about the symbol rate.
What you hence are looking for is a symbol rate recovery algorithm, to know how often a symbol comes through, and a timing recovery algorithm, to tell you when exactly to look for the symbol. (The terminology I use here isn't that common, often "timing recovery" means both, getting the rate of symbols, and the exact temporal point.)
There's quite a few of these! The "classical" symbol rate recovery one for linear modulations (PSK, ASK, QAM, that kind of thing) is Oerder & Meyr [2], which exploits the fact that squaring the signal will introduce spectral components at multiples of the symbol rate. From there it's just a frequency estimation problem.
Problem is that it assumes a) sufficient SNR so that squaring doesn't make noise more powerful than your desired tones, and b) that the pulse shape after your receive filter adheres to the first Nyquist criterion. b) is a real problem, because you don't know the transmit filter, and systems usually only adhere to Nyquist I after the matched filter in the receiver (and that's matched to the unknown transmit filter).
There's some error considerations (and you can probably really work without exact knowledge of the transmit filter), but if you use a blind equalizer to bring your pre-channelized signal back to a white spectrum, you already have a starting point for your filter assumptions.
If you know anything about the transmitted signals (or dare to hypothesize and experiment), you can be much better. For example, if you know (or assume) the transmit signal is an OFDM variant, there's methods for that. For example, take this GNU Radio "Google Summer of Code" project [5], where a student designed a whole system to in general do something similar to what you describe you've done so far, and if it detects OFDM, to give you OFDM symbol length, bandwidth/rates, prefix dimension, and timing information.
[1] F. Jondral, "Automatic classification of high frequency signals," Signal Processing, vol. 9, no. 3, pp. 177–190, 1985; excerpt
[2] Oerder, M. ; Meyr, H.: Digital filter and square timing recovery. In: Communications, IEEE Transactions on 36 (1988), Mai, Nr. 5, S. 605 –612. http://dx.doi.org/10.1109/26.1476 . PDF – DOI 10.1109/26.1476. – ISSN 0090–6778
[3] PROCITEC: go2MONITOR, go2DECODE: Software for automated transmission classification and payload information extraction, https://www.procitec.de/en/products/
[4] deepsig: OmniSIG Sensor: detection and classification of RF emissions across large spectrum bandwidths in millisecond, https://www.deepsig.ai/omnisig
[5] Sebastian Müller: The Inspector. gr-inspector: A Signal Analysis Toolbox for GNU Radio, https://grinspector.wordpress.com/ . Demonstration video, Source code
