I need a sanity check. I think I understand the following, but I just can't get it to work.
Suppose that I calculate a trifocal tensor $T$ from three sets of corresponding points in three images (call the sets $x$, $x'$, and $x''$). From $T$, I calculate three cameras, $P$, $P'$, $P''$, using Algorithm 15.1 in Hartley & Zisserman's Multiple View Geometry in Computer Vision (Algorithm 14.1 in this PDF copy), so that all three cameras are in the same projective frame.
Now I triangulate two points $x_i$ and $x'_i$ from the first two sets, resulting in a 3D point $X_i$. As expected, $PX_i = x_i$ and $P'X_i = x'_i$, up to a scale factor.
Here's the question: since $P''$ is in the same projective frame, if I project $X_i$ with $P''$, (i.e., $P'' X_i$), shouldn't I get $x''_i$ (up to a scale factor)?
It seems like it should work (otherwise, what good is it for $P''$ to be in the same projective frame?), but I've been over my code dozens and dozens of times, and I can't find the bug. I just need to know if my understanding is correct, or if I'm missing something fundamental.