Intuition about tag pose estimation accuracy

Question

I have some difficulties to have a correct intuition about expected pose accuracy when dealing with planar pose estimation (e.g. pose estimation from AprilTag detection) and perspective camera model.

Assuming ideal conditions (e.g. simulated data), I consider also that the tag corners can be extracted with a good accuracy, e.g. < 1px.

1. Given two physical tags projected onto the image plane with the same size, oblique view (tag parallel to the image plane), can we assume the expected pose error is the same regardless of the working distance?

• Since for tag detection, what matters is the size of the tag in the image, and for pose estimation what matters is precise corners location extraction, I want to put a minimal tag size in the image to choose other factors like physical tag size, camera focal length, working distance, ...
• If 1. is true, this means that for two tags projected with a 100 px size in the image, if we have a translation pose error of 1mm at 1m, we could assume to have 1mm translation pose error at 2m if we choose appropriately the physical tag size to ensure a projected tag size of 100px.
• My intuition for 1. is that it is not true, and somehow the translation pose error is dependent to the working distance.

2. For a fixed working distance, if a tag is projected onto the image plane with a 100px size, what is the impact of the projected tag size wrt to the translation pose error?

• For instance, if the size of a tag is 100px in the image and the translation pose error is 1mm, if we double the size of the tag in the image to be 200px by playing with the focal length or by increasing the physical tag size, can we expect to reduce the translation pose error by the same factor (i.e. 0.5mm)?
• For sure bigger tag size should improve the accuracy of the pose estimation, but I am not sure it scales like that.

---

By pose estimation, I mean the translation ($t_x, t_y, t_z$) + orientation ($3 \times 3$ rotation matrix) for rigid body transformation. Also called PnP pose estimation, like pose from homography for planar object or EPnP by Lepetit et al.

For perspective projection:

• normalized image coordinates $$ x = \frac{X_c}{Z_c} \\ y = \frac{Y_c}{Z_c} $$
• to the image coordinates $$ u = x \times f_x + c_x \\ v = y \times f_y + c_y $$

For focal length in pixel, I have: $$ f_xpx = \frac{f_xmm \times \text{image_width}}{\text{sensor_width_mm}} $$

---

This intuition would be very useful to accordingly choose the physical tag size, the focal length, the image resolution to ensure or at least maximize the probability to have a given translation pose error.

Answer 1

The size of the tag is an important parameter. But it is a secondary parameter. The key parameter that we are looking at here (and one that is not easy to estimate) is Signal to Noise Ratio (SNR). So, very very briefly: The bigger the tag, the more data we obtain from the tag, the better the position of the estimate will be.

But, the "size of the tag" as perceived through a camera is an entirely relative concept. If you take a picture of a lighthouse that is far from the coast with a web cam at 320x240, you will see (for example) a flickering pixel patch of size $6 \times 2$. You can attach a lens to the camera or crank its resolution up. Both of these solutions, will allow you to image the lighthouse much better. If either of these options are not available then, there is a maximum distance (which puts a limit on the perceived size of the lighthouse) within which the lighthouse is sufficiently sampled.

So, the perceived size of the tag is only relevant with respect to the SNR. In low noise, small patches will be extracted more accurately and vice versa.

A discussion about accuracy must go through the way the detector derives the position of the tag.

Very briefly: There are two parts to it, detection (What pixels make up a "tag" in an image?) and decoding (Which tag(s) appear(s) in an image?). We leave decoding aside for a minute. The first step in detecting where the tag is is the computation of the gradient (magnitude and direction). The second step is the grouping of similar value gradients. Gradients (slopes) of similar value help in picking out lines. So each one of these groups (potentially) contains a line. We take the $(x,y)$ data of those groups and we use least squares to fit lines. At that point detection concludes. If you do this for all connected components, you will get back a data structure of connected lines that describes where the tag is (within the camera's field of view) and its orientation. There are some hints that we take from detection that are then used to decode the payload of the tag but this highlights why the tags are composed of black and white patches.

The simplest extreme counter-example that shows that the size of the tag is not really the issue here is this: Put a large enough tag in front of the camera and turn the contrast all the way down. This "kills" the gradient which is the main way by which the vision algorithm makes sense where the tag is. If the image is a grey fog, no tag will be detected because the gradient will return zero groups of pixels.

Conversely, suppose that the tag is of sufficient size, the contrast is optimal, the lighting is perfect, but, the code is amongst a huge herd of zebras chilling out in a field. That is, the background is now overwhelming the sensor with a huge amount of possible gradients that form lines that COULD lead to a tag just by chance.

So, the size of the tag is relevant with respect to the disturbance present in the image.

The noise, spreads out the gradient magnitudes and directions, which introduces uncertainty to the least squares of line fitting, which introduces uncertainties to the determination of position and orientation of the tag. The tag is black and white because this maximises the response of its detector. The gradient. It improves its chances of picking it out in the image.

We can derive some simple limits here of course. For example, you need at least 2 points per line segment of a tag so that all line segments that compose the tag have a chance of making it through the detector succesfully and the whole tag gets a chance to be recognised. So, the smallest perceived size for the tag is its size as if it was a set of pixel patches times 2. If you have a 4x6 tag (that is 4x6 black and white pixels), it needs to be imaged at least at 8x12 pixels. If not, you don't have 2 points per line to define it.

Then, what is the proportion of light that falls on the sensor that is not due to the tag? The variation of that light, blurs the gradient, which blurs the position of the recognised lines. Once the gradient is blurred the position is uncertain. The point is this, if you want to keep the ratio of signal created by the tag to signal NOT created by the tag constant, you need to vary the size of the tag so that you start enjoying the benefits of least squares that follows the gradient. More data, more chances of fitting the true line. Varying the size of the tag means either making it bigger, imaging at closer distance, imaging it through a larger magnitude lens, increasing the contrast, etc.

You might notice that this is not addressed in the original paper too. The authors provide results for recognition error rate versus distance and view angle but these are based on simulated data and also, they are not absolute (they cannot be absolute). They are comparative. They make sense with comparison to another method. (And also, they include the decoding of the tag content which puts additional resilience to the SNR we talked about before. While we are at this, you might want to see section "VI.b. Localisation Accuracy" and maybe section "III.Detector" which describe how the code is picked up in more detail than I go into here)

So, having said this, let's revisit the question:

"Assuming ideal conditions (e.g. simulated data), I consider also that the tag corners can be extracted with a good accuracy, e.g. < 1px."

If the conditions are ideal, the gradient-->least squares works perfectly and the pose is recognised to the limit of numerical accuracy (and image resolution).

Given two physical tags projected onto the image plane with the same size, oblique view (tag parallel to the image plane), can we assume the expected pose error is the same regardless of the working distance?

Yes, provided that the tags are within the limits of detection we talked about before. Side note: The tag that provides the most information to the gradient-->least squares part of the detector will have the best accuracy (between the two we are looking at).

For a fixed working distance, if a tag is projected onto the image plane with a 100px size, what is the impact of the projected tag size wrt to the translation pose error?

The question is ill posed. The size of the tag is only half the story. What else is going on around the tag?

I think that it will be very difficult to arrive at a quantitative estimation of this in a way that it would drive the selection of the tag size without some sort of experimentation or some sort of regularisation or control for some of these factors.

I hope this helps.