I am going to break down this answer into two main parts, one for a step-by-step computation of the STFT, and the other in regards to how to compute the MFCC's, once you have the STFT.
STFT: Let us say you wanted to compute the STFT of your signal $x[n]$, of total length $N=100$, and block size $M = 10$, with $50$% overlap, and using a hamming window. I will give you a step by step, (and redundant) flow, to get you started:
- Take the first $M=10$ samples out, that is, samples $x[0], x[1], x[2] ... x[9]$.
- Window them, by multiplying them, element by element, with your window coefficients.
- Take the DFT of this result.
- Take the next $M=10$ samples out, that is, samples $x[5], x[6], x[7], ... x[14]$.
- Window them, by multiplying them, element by element, with your window coefficients.
- Take the DFT of this result.
etc, etc. You get the idea. Let us say that you took a (same size) DFT in the above example. Then, you will end up with a $10$ x $19$ complex matrix. If you then simply take the absolute magnitude of the above matrix, you will end up with a real matrix, corresponding to the absolute magnitude of your complex matrix. This is called the Short-Time-Fourier-Transform, or STFT. We will use the STFT as the input into computation of the MFCCs.
Before that, let us get some formulas and terminology straight: Let us call our STFT matrix $S$. This matrix is composed of $B=19$ blocks, or columns, and $K=10$ frequency bands, or rows. To refer to a time-frequency point in $S$, we will say $S[k,b]$, where $b \in 1, 2, ... B$, temporal blocks, (columns) and $k \in 1, 2, ... K$ sub-bands, (rows).
MFCC:
Once you have the STFT computed, you can go ahead and use that as a stepping stone for computing the MFCC's. Regarding your question, you seem to be confusing the time domain windowing done on each block for the STFT, and the frequency domain windowing done in the MFCC. They are completely separate and unrelated.
Let us say that you computed your STFT using a hamming window, and you have a $K=500$ x $B=1000$ real matrix. Thus using our prior terminology, you have a real matrix $S_{K \text x B}$. Thus, you have $500$ sub-bands, (frequencies), and $1000$ blocks, (time).
Let $Y_{b}[k] = |S[k,b]|^2$. That is, we are letting $Y_{b}[k]$ represent the power-spectral density of a particular column/block of the STFT matrix $S$. Obviously, each $Y_{b}[k]$ is a $500$ x $1$ column vector.
Clearly, we have $1000$ $Y$'s. That is, we have $Y_{1}[k], Y_{2}[k], Y_{3}[k], ... Y_{1000}[k]$.
Now, you want to compute the MFCC of each column. Take the first column. This is $Y_{1}[k]$. Forget about all the other columns, and forget about the hamming window. The first column is now used in your MFCC algorithm. The first step in MFCC was getting a PSD column, which you have, because we have $Y_{1}[k]$.
The second step is the computation of, say, 10 power estimates. This is computed by windowing this column, by 10 overlapping windows, as per the mel-scale, like so:

Note how each (triangular) window is mostly zeros. So now you literally do a dot product of each window, with $Y_{1}[k]$. You will get $10$ numbers, for this column. Now take the log of those $10$ numbers. Now take the DCT of this result. Congratulations! You just computed the MFCC for this column. You just computed the MFCCs for $Y_{1}[k]$.
Now, go back, and repeat for all the other $Y_{2}[k]$. Then repeat for $Y_{3}[k]$, etc.