1
$\begingroup$

Problem:

  • I need to measure a experimental data set $y(x)$.

  • However, the experiment is extremely time-consuming and I don't have the luxury to have linearly spaced sampling point $x$.

  • I need the density of sample points to be proportional to the first order derivative of the data $y^{'}(x)$.

Progress:

  • first measure $y(x)$ once with uniform x as a test run, then compute $y^{'}(x)$.
  • Then fit $y^{'}(x)$ to $pdf(x,\alpha, \beta, ... )$.
  • Then I use the parameters to calculate new data points with inverse CDF $X=icdf[linspace[pdf(min),pdf(max),N],\alpha, \beta, ...]$

The problem is, this method only apply to data sets that is distributed in common probability distribution functions.

Is there any way to make this work with arbitrary data ?

Thanks in advance !

$\endgroup$
1
  • $\begingroup$ "I don't have the luxury to have linearly spaced sampling point x." Meaning you don't have storage space to keep that much data, or the actual measurement itself is expensive? $\endgroup$
    – TimWescott
    May 21 at 15:01
1
$\begingroup$

A method that I use very often to approximate the CDF is based on the observation that given a set of $N$ samples, the CDF at the point x the fraction of the points that are smaller than $x$. So if have a vector of sorted samples $S$ you could approximate the CDF as $cdf(x) = i/N$ if $S_{i-1} < x < S_i$.

A simple way to plot the CDF would be

plot(sort(S), (1:N)/N)

Then you can approximate the derivative by a difference equation. You could use a binary search to find the region of the vector $S$ where $x$ lies, say $idx(S, x)$, then you could compute something like

$i_1 = idx(S, x - \Delta)$, $i_2 = idx(S, s + \Delta)$ and use $(N \cdot (S_{i_2+1} - S_{i_1})) / (i_2 - i_1 + 1)$.

A different way to choose the anchor points would be, let $i = idx(S, x)$ and compute $(N \cdot (S_{i+k} - S_{i-k})) / (2k+1)$

And you can fit use a polynomial fit if you need a better accuracy where the CDF derivatives varies rapidly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.