I am going to provide some initial information as simply as I can that should be helpful in the OP's attempt to get a basic understanding of control systems. Further study in Laplace transforms (and z-transforms for discrete-time systems) will be helpful in these frequency domain modelling approaches to control systems.
First understand the concept of a transfer function as the frequency response for a basic two port (one input / one output) linear system; the transfer function defines the magnitude and phase versus frequency of the output compared to the input. This is often given in terms of the variable "s", but to avoid going down that path I am going to present it as a function of frequency only ($H(f)$). Below shows an example system with a plot of the magnitude and phase versus frequency as given by the $H(f)$ equation. This shows us for any particular frequency at the input, what the amplitude of it will be at the output and what the phase shift will be between the input and output: every frequency will be transformed by the system by changing its amplitude and phase shift only, as long as it is a linear system no other frequencies will be created so this is a nice compact description that we will ultimately use to determine stability in a control system.
Now consider the simplest case of a system with feedback as diagrammed below with input given as $X$ and output given as $Y$ with the output fed back and subtracted from the input:
The relationship between $Y$ and $X$ is easy to show with simple algebra, with our goal to describe a new transfer function of this new closed-loop system. The new transfer function would be given as $Y/X$ (output divided by input). I don't show it below to not clutter the simplicity of the equation but $X$ and $Y$ are also functions of frequency as $X(f)$ and $Y(f)$, so would be the Fourier Transforms of time domain signals:
$$(X-Y)H(f) = Y$$
$$\frac{Y}{X} = \frac{H(f)}{1+H(f)}$$
The above equation would adequately describe our new closed-loop system as long as it follow the Nyquist Stability Criterion. To really show this completely to cover all cases, the OP needs to first understand the Laplace Transform and the generalized transfer function as a function of s (as in $H(s)$), but here in this simple introduction we can reduce it to a condition that is covered in most cases: that is if the gain of the system given by $|H(f)|$ is less than one before the phase of the system given by $\angle (H(f))$ exceeds 180°, the system will be stable and will have its own closed loop transfer function given by the expression above. We can see this through an iteration of a simple case with H(f) = -.8 at a particular frequency (gain = 0.8 and phase = 180°).
Consider the feedback system as an iteration with the input starting at 1: the output will be -.8, which fed back to the input as a subtraction will add 0.8 to the input, resulting in 1.8 at the output. This is then multiplied by -.8 resulting in 1.44 being added to the input resulting in 2.44 at the output...
With closer inspection we see that this is a geometric series:
$$1 + \alpha + \alpha^2 + \ldots = \sum_{n=0}^\infty(\alpha^n) = \frac{1}{1-\alpha}$$
Which converges only if $-1 < \alpha < 1$.
Where here $\alpha$ was 0.8 (due to the negative feedback in the loop) which would converge to $1/(1-.8) = 5.
If the gain of $H(f)$ exceeded one with the phase 180°, we would see how the same geometric series would not converge, as an example of an unstable loop.
This gives a very basic simplified explanation to help aid in an intuitive understanding, but further proper study would be to understand the Laplace Transform and complex systems, and the meaning of poles and zeros in the s-plane, and how a stable system must have all of its poles in the left-half plane, and ultimately the full details of the Nyquist Stability Criterion.
