2
$\begingroup$

I am running into a problem where I have a control system $S[t]$ that takes a control $C[t]$, so that $$S[t+1] = H(C[t,t-1,...], S[t,t-1,...])$$ the response of the system is the history of controls and responses.

I am looking for a way to control $C[t]$ to optimize the value of $S[t]$ at any point in time.

What is the literature on this problem called? Where do I start looking? Thanks.

$\endgroup$
1
  • $\begingroup$ Your Question should be more accurate, I think that I can help you. $\endgroup$ Feb 10, 2020 at 15:11

2 Answers 2

2
$\begingroup$

This looks like it might be able to be written as a dynamic programming problem. Bellman's equation looks very similar to the way you have expressed your control problem.

$\endgroup$
2
  • $\begingroup$ yes, RL is the only solution I can find, but I think the problem is simple enough that there may be some theoretical control theory for it $\endgroup$
    – thang
    Nov 8, 2018 at 12:07
  • $\begingroup$ @thang the problem you have stated is not necessarily a reinforcement learning (RL) per se but by definition is a delayed control problem as I explain in my answer. $\endgroup$
    – kbakshi314
    Apr 4, 2021 at 23:32
2
$\begingroup$

The problem stated by the OP is non-standard since the dynamics are time-delayed. It is made further non-standard due to the fact that the state transition dynamics depend on prior states, thus making it a non-Markov system.

Such systems have been treated using classical control, nonlinear feedback control and reinforcement learning (RL) although almost all RL algorithms are designed for Markov processes which do not admit time-delays.

While the first reference linked in this answer treats a system closest to the one posed in the OP, a good suggestion is to start with an approximation of the high-fidelity transfer function in the z-domain and design the controller in the z-domain since delays can be modeled exactly (provided one uses a high sample rate) in that domain. In the z-domain design approach, you can start your design process neglecting delays and introduce them gradually in order to see how delay degrades the performance of the closed-loop system.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.