The OP's question reveals many mis-understandings of the issues.
The convolutional code in question is a nonsystematic $[2,1]$ convolutional code with constraint length $K = 3$. The specification is incomplete but the complete specification (we also need to know what the generator polynomials are) can be deduced from the state transition diagram that the OP has provided. This particular convolutional code is used as the first example in almost all texts on coding, and so the code is sometimes referred to as the ad nauseam code. Let's pick apart what things mean. There is 1 input stream of bits entering the encoder during each clock cycle and 2 streams of bits leaving the encoder (in parallel, on two separate wires) during the clock cycle. The encoder itself contains one shift register (not $3$ as the OP thinks) of length K-1 = 2 meaning that it is capable of storing two bits. The encoder also contains XOR gates for computing the output bits from the encoder input bit and the shift register contents. The bits stored in the shift register are the immediate past input bit and the input bit immediately previous to the immediate past input bit, that is, the input bit from two clock cycles ago. Thus, this single two-bit shift register can be in one of $2^{K-1} = 4$ states rather than three shift registers somehow managing to be one of only $4$ states which has puzzled the OP (and would puzzle anyone else too). At the end of the clock cycle, the current bit enters into the shift register, becoming the "immediately past input bit" for the next clock cycle while the bit that served as the "immediately past input bit" for the current clock cycle moves over one place to become the "input bit two cycles ago" for the next clock cycle. At the start, the shift register is initialized with two $0$s in it. That is, the encoder is in state $00$ at the beginning of the encoding process.
During any clock cycle, K = 3 bits viz. the current input bit, the immediately past input bit and the input bit from two clock cycles ago, are available for computing the 2 output bits,. The state transition diagram of the encoder is a directed graph whose vertices labeled with the state that the encoder is in, and the label of the directed edge from one state to another state denotes "current input/output bits". For example, if the current state is $00$ and the input is a $0$, the encoder outputs $00$ and remains in state $00$. The edge showing this transition is thus labeled $0/00$ and the edge leaves and enters state $00$. On the other hand, if the input is a $1$, the output is $11$ and the encoder moves from state $00$ to state $01$. This the directed edge from $00$ to $01$ is labelled $1/11$. We also deduce from this that whomsoever drew the logic map for the encoder had the input entering the encoder from the right and the shift register shifting leftwards. If the input bits had been entering from the left and the shift register shifting rightwards, the next state would have been labelled $10$; you pays your money and you makes your choice.
Note that there is no edge from state $00$ to state $11$ or vice versa; there is no way that this particular encoder shift register can make a transition from state $00$ to state $11$ or vice versa in one clock cycle. Note also the symmetries in the state transition diagram. The two edges leaving each state have complementary input labels (obviously, if you think about it a bit, pun intended) and complementary output labels too. The two edges entering each state also have complementary output labels (but the same input label; can you see why?).
So, how does the encoder actually work? Well, if the input data stream is
$$ \cdots d[n-4]\quad d[n-3] \quad d[n-2] \quad d[n-1] \quad d[n] \quad d[n+1] \quad \cdots$$
then at the $n$-th clock cycle, the encoder shift register contains $d[n-2]$ and $d[n-1]$ as indicated below via the box, and $d[n]$ is the current input to the encoder shift register.
$$ \cdots d[n-4]\quad d[n-3] \quad\bbox[yellow,5px,border:2px solid red] {d[n-2] \quad d[n-1]} \quad d[n] \quad d[n+1] \quad \cdots$$
During this clock cycle, the encoder produces two code bits
\begin{align}
c_i[n] &= d[n]\oplus d[n-1] \oplus d[n-2],\\
c_q[n] &= d[n] \oplus d[n-2].
\end{align}
The shift register then shifts, and during the $(n+1)$-th clock cycle the register contents are as shown below and the input data bits is $d[n+1]$.
$$ \cdots d[n-4]\quad d[n-3] d[n-2] \quad \bbox[yellow,5px,border:2px solid red] {d[n-1] \quad d[n]} \quad d[n+1] \quad \cdots$$
The generator matrix for this code is
$${\bf G} = \big[ 1 + x + x^2 \quad 1 + x^2\big]$$
meaning that if we regard the input data stream as a series
$$d(x) = d_0 + d_1x + d_2x^2 + \cdots,$$ then the encoder output streams can also be regarded as series
\begin{align} c_I(x) &= (1 + x + x^2)d(x)\\
&= d_0 + (d_1 + d_0)x + (d_2 + d_1 + d_0)x^2 + \cdots\\
& \qquad \quad \cdots + (d_n + d_{n-1} + d_{n-2})x^n + \cdots\\
c_Q(x) &= (1 + x^2)d(x)\\
&= d_0 + (d_1)x + (d_2 + d_0)x^2 + \cdots\\
& \qquad \quad \cdots + (d_n + d_{n-2})x^n + \cdots
\end{align}
or, in short,
$$d(x){\bf G} = \big[c_I(x) \quad c_Q(x)\big] = \big[(1+x+x^2)d(x) \quad (1+x^2)d(x)\big].$$
Dedicated DSPissers will replace $x$ by $z^{-1}$ and say "Hey, you are telling me that $c_I(z) = (1+z^{-1} + z^{-2})d(z)$ and $c_Q(z) = (1+z^{-2})d(z)$, and I know that multiplication in the $z$ domain corresponds to convolution in the time domain, and so why don't you name these codes as convolutional codes and save a lot of explanations on dsp.SE?" and that's exactly what it is: the encoder output bits are obtained by convolving the encoder input sequence with the generator polynomial sequences.
I've seen on the internet something weird that the transition table for convolutional encoder $K=3$
is: transitionMatrix = [0 1 1 55; 0 0 1 1; 55 0 55 1; 55 0 55 1]; it was written that value $55$ is value of STATE HISTORY .
I didn't understand this matrix and how this matrix describe the state machine that I attached the photo above for convolutional encoder $K=3$
I too have no idea what that gobbledygook means.
In a comment, the OP asks about implementing a transition matrix. There may be a language barrier here in communicating with each other. I don't know what exactly implementing a transition matrix means or why one would want to go about implementing one. I know about people implementing the encoder from the state transition diagram, whether in hardware with flip-flops and gates, or via FPGAs, or in MATLAB or Python or C++ etc. When the generator matrix is not known, it takes a little analysis of the state transition diagram to figure out what the generator polynomials are, but once that is done, implementing the encoder is quite straightforward. Another way of looking at it is that the encoder implementation implicitly implements the transition matrix in that the transitions that the encoder makes follows the rules laid out by the transition matrix.
