Is it as simple as needing at least as many independent measurements as you have things you want to estimate? If you have only four measurements (color patches), but need to find nine entries for the 3×3 color correction matrix, that seems like an underdetermined system (in linear algebra terms).

Remark: Answer by the comment of @Peter K. above.