2
$\begingroup$

Given the list of data points (normalized in [0,1] range), I plot the histogram of values and compute percentiles (shown as x ticks).

value distribution of data

How to find a transformation of data values so the histogram is approximately uniform. Which would, in turn, make percentile values also uniformly distributed.

| improve this question | |
$\endgroup$
2
$\begingroup$

Hi: You can calculate the empirical cumulative distribution of the data. By this, I mean, given some observation in the sample, $x_i$, calculate $P(X < x_{i})$ by calculating the proportion of observations that are less than $x_{i}$ (i.e. the percentiles ). Then, do this for all the $x_{i}$ so that you have the cumulative distribution of the $x_{i}$.

Then, $P(X < x)$ is uniform for a given value of $x$.

In fact, it seems like you already did this but the percentile values should be on the vertical axis and the values of the data should be on the x-axis.

Note that page 14 of this PDF explains the concept more clearly than I have.

Example Implementation

Below is a quick-and-dirty attempt to illustrate this answer. The image below shows the original histogram of the Gaussian, the empirical cumulative distribution function of that data, and then the histogram of the converted data.

Plots of example implementation

R Code Below

par(mfrow=c(3,1))
# First, generate some Gaussian numbers.
gaussian <- rnorm(1000,0.0,0.05)
gh <- hist(gaussian, breaks=1000)

empirical_cumulative_distribution <- cumsum(gh$counts)/1000

plot(gh$mids, empirical_cumulative_distribution)


uniformize <- function(x) {
  ans_x <- x
  for (idx in seq(1,length(x))){
    max_idx <- max(which(gh$mids < x[idx]))
    ans_x[idx] <- empirical_cumulative_distribution[max_idx]
  }
  return(ans_x)
}

uniform2 <- uniformize(gaussian )
hist(uniform2, breaks=100)
par(mfrow=c(1,1))
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Hi Peter: you're making me look bad :). thanks. $\endgroup$ – mark leeds Sep 14 '19 at 0:53
  • $\begingroup$ @Peter K: Is there something that shows you to do links so that I can do what you did as far as pointing to the actual pdf rather than typing the link ? Or, is there a way for me to see your latex so that I can see what you did. I have been told that I should learn this and I should !!!!! Thanks. $\endgroup$ – mark leeds Sep 14 '19 at 0:58
  • $\begingroup$ To inline links, just select the text you want to add the link to and hit the link/chain icon in the top of the text editor. I tend to prefer linked text to raw links, so I change it when I see it (and can be bothered). $\endgroup$ – Peter K.♦ Sep 14 '19 at 1:03
  • $\begingroup$ @Peter K: I forgot that that was "my answer" so I was able to just hit the edit button to look at what you did. I learned a lot and will use it in the future when I want to do that better way of linking. Thanks. $\endgroup$ – mark leeds Sep 14 '19 at 6:26
  • 1
    $\begingroup$ @Peter K: R is s vast so it has many "levels" of users-developers. If you want to up your R game, get Hadley Wickham's "Advanced R book" but not the most recent edition. The recent edition I think focuses on Hadley's tidyverse ( which I'm confident is fine if you're into that tidyverse material ) but I have the first edition and that focus was on advanced language concepts. A really nice exposition. $\endgroup$ – mark leeds Sep 15 '19 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.