4
$\begingroup$

I've a scatter plot with x- and y-axis.

Now I want to calculate a rectangle which displays where 80% of the points in this plot are.

I'm not very familiar with statistics, so I don't know the method I have to use. Read something about percentiles. Is that correct?

I know I have to calculate where 80% are on the x-axis and also where 80% are on the y-axis and then I got my rectangle corners, but how do I calculate the 80%?

Sample data:

Sample data

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Can you post a picture of the data, along with where you'd expect the rectangle to be? I think you need more constraints too - I'd expect there to be several places you could put rectangles which encompass 80% of most datasets... but yours may be distributed in a special way. $\endgroup$
    – Martin Thompson
    Commented Mar 27, 2013 at 10:41
  • $\begingroup$ Sure, but it's just an example. I want to determine where most of the points are in the plot: see Plot $\endgroup$
    – Jim Panse
    Commented Mar 27, 2013 at 11:03
  • 4
    $\begingroup$ There are multiple solutions- i.e. multiple boxes- for any data set that will capture 80% of the data points. Can you constrain the problem any more? $\endgroup$
    – Jim Clay
    Commented Mar 27, 2013 at 15:21
  • $\begingroup$ Ok, i want to help the user to easily recognize where the density of the point cloud is very high and therefore draw a rectangle around them. Because the plots have mostly a dense pointcloud and multiple outliers (like in the example), i want to build the "80%-of-the-points-rectangle" around the cloud, containing the dense pointcloud and including 80% of the points in the plot where the overall density in the rectangle is at its highest. $\endgroup$
    – Jim Panse
    Commented Mar 28, 2013 at 8:08
  • 1
    $\begingroup$ @JimPanse You need to specify how you want to choose between multiple different 80%-of-the-points-rectangles for the same point cloud. To explain what I mean, imagine a simple 10x10 grid of points (100 points). The rectangle could contain all columns except the last two, or the first two, or the first and last, or all rows in the same manner, giving the same solution. If you constrain this to "the 80% rectangle with its center nearest the center of the point cloud" then there are still two solutions in that case. In less simple cases you would also probably want the smallest rectangle. $\endgroup$
    – Sparr
    Commented Mar 28, 2013 at 14:22

2 Answers 2

Reset to default
2
$\begingroup$

Your question seems to imply that there will always be just a single cluster of points. If this is the case, I would go with a fairly simple heuristic like the following-

Calculate the mean and variance of the point positions in both the x and y directions. The mean is the center point of your box, and you can use the ratio of the x/y variances to determine in which directions the box should grow faster. For instance, if the x variance is "1" and the y variance is "2", I would have the box expand twice as fast in the y direction.

Now simply expand the box until you have at least 80% of the points contained.

If there is ever more than one cluster of points you may need to use a more sophisticated solution that uses clustering algorithms to identify the clusters and a box for each cluster.

Improve this answer
$\endgroup$
1
$\begingroup$

First of all, I donot think you described your problem properly, because you might find multiple 'boxes' containing 80% of points. I call your box of interest as BOI in the rest of my post.

I've no idea what you gonna to do with this BOI, but I guess you are trying to do some data visualization or find criteria to differentiate outliners. There are many more professional ways to visualize scatter data points within a certain confidence interval.

In short, these algorithms normally do the following things 1) estimate the statistics of the given sample data, eg. mean and standard deviation 2) compute the confidence interval of a certain level, according to your prescribed data distribution or estimated distribution 3) plot the contours of the found confidence interval on the scatter plot

Although the rectangular region of a confidence interval sounds simple, it is clearly less accurate than than other shapes, e.g. a eclipse used in guassian distributed data. So if you are writing a scientific report, please follow these existing methods.

In case you insist to find the BOI, you can following the similar steps with additional considerations on defining the best BOI of all found BOIs. Why? Because you shall see multiple BOIs that containing 80% of data points. To be frank, finding the best BOI is a task that is more difficult then it looks like. However, several things are clear here: 1) the mean point of the data set should be in the best BOI 2) we should define what is your best BOI (I think the BOI with the minimum variance should be the best)

The following described method is actually something like the histogram analysis. To find the best BOI, you shall put your data points into bins, and allows us to compute histogram later. Note the bin width defines the resolution of our method. If it is too big, for example, as wide as your data point span, then you will never find a so-call 80% BOI. If it is tot small, for example, the finite precision of your machine, then you might take hours to find the best BOI. In general, simply pick something moderate is OK (or you can implement adaptive methods to pick the width and height of the bin for you data).

Assume you pick the bin size. Then you simply compute the histogram of your data $T=\textrm{Hist}(Data|bin)$, where each bin contains an integer number of points. You should have a 2D histogram if your data is 2D. Once you get this histogram, retrieve its height $h$ and width $w$, then do the following things, you shall find the unique BOI defined as the quadruple of $(x,y,W,H)$, where $x,y$ defines the upper-left point of the $W\times H$ BOI.

  1. Construct the integral image of this histogram $\textrm{II}(T)$(how to do? see details in wiki. This fast method will allow you to compute the number of points within an arbitrary BOI for only three operations.)
  2. Initialize $v_{min} = realmax$
  3. For each possible BOI size $W\times H$, you retrieve the number of data points of $W\times H$ BOI from $\textrm{II}(T)$
  4. Eliminate all candidate BOIs without the mean point
  5. Eliminate all candidate BOIs with less than 80% points
  6. Compute the variance of data points within each candidate BOI
  7. Find the BOI with the minimum variance as $v$ of size $W\times H$
  8. If $v<v_{min}$, then update $v_{min}=v$ and record the quadruple of this BOI.
  9. Repeat step 3-8 until you search all possible sizes

In the end, this should give you a unique BOI with the minimum variance. Depending on the number of data points and the bin size, you might expect the above algorithm takes a reasonable long time to get your the best BOI.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.