I'm currently studying the use of Kalman filters for estimating linear systems. My current State Transition Matrix (STM) is the identity since so far I've been dealing with non time-varying systems. However my next step is to extend to time-varying systems and in this case there is no information on how the system varies and so the STM is unknown. And as of my testing so far a wrong STM causes a ton of errors on the Kalman Filter estimations. So my question is if there is a method for circumventing this, be it by updating the STM on the fly or some kind of way of updating the Process Noise Covariance Matrix in order to account for that?
Unfortunately, in Classical Kalman filter applications you need to have a guess of your state transition model governing the states. Classical choices for such models for time-varying systems are the random walk model, or its generalization, a first-order Markov model. If do not have a clue about which state model you can assume, you can try to extract a trend component from your state time series (via EMD or a moving average filter, for example), and plot it to have an idea of how the first-order moment of your states vary with time, for example. If it decays as a power law with $n$ (i.e. $E[x(n)] \approx \alpha^n$) until a final value, it may be an indication that a first-order Markov model with pole equal to $\alpha$ can serve as a state transition model.
Otherwise, you can treat the transition model for your states as a black box and consider techniques such as the Unscented Kalman Filter, for example.
In the Bayes Net Toolbox (BNT) for Matlab, there is an example under Kalman Filter where the EM Algorithm is used to estimate the State Transition Matrix (and the rest). It is a 2 pass batch algorithm.
The only problem is that BNT was written for Matlab V5 and the github version has some updates but under 2017a, there are a few problems, nevertheless I found it useful.
The other limitation is that it is for a strictly Kalman Filter, not an EKF, UKF, or particle filter.