I'm not sure what the freak out about causality or lack thereof is about. You can approach this problem just by thinking about linear algebra. $L$ is a linear transformation. Applying $L$ to the input is just matrix multiplication. So we have $$Lx = y$$ If $x$ is an impulse then it's just picking out a column of $L$, so the columns of $L$ are the impulse responses. Of course, 3 input-output pairs is not enough to completely determine $L$ as a 5x5 matrix.
Let's consider what time-invariance would mean from this perspective. If a transformation is linear and time-invariant then it's impulse response always has the same shape and is only shifted in time by the same amount as the input impulse. So let's say the impulse response for $L$ is 0 1 2 3 0 centered on top of the input impulse (and thus non-causal). The matrix for a linear time-invariant $L$ would then look like:
$$L = \left(\begin{array}{ccc}
2 & 1 & 0 & 0 & 0 \\
3 & 2 & 1 & 0 & 0 \\
0 & 3 & 2 & 1 & 0 \\
0 & 0 & 3 & 2 & 1 \\
0 & 0 & 0 & 3 & 2
\end{array}\right)$$
So, to answer the first question, you just need to build enough of two columns to see that they are different to disprove time-invariance. A direct way to do this is to assume it is time-invariant and derive a contradiction. However, to show that it is time-invariant requires more information, i.e. it requires completely specifying the matrix. If it is not time-invariant, then there is a potentially different impulse response for each sample, not a single one, as others have mentioned.
Ultimately, as others have alluded to, we can't actually know whether a linear system is time-invariant or what it's impulse response is just by looking at short input-output pairs without more information. For all we know, $L$ is a 1,000,000 wide FIR filter or even an IIR filter that just happens to be 0 near the middle. Or it appears time-invariant so far, but it changes in the next sample. In general, we have to use multiple hypothesis testing to pick what the evidence best supports. Probability theory is a crucial part of signal processing.