I have two vectors of different length, each vector contains similarity scores. I need to plot the probabilty density function of the scores in both vectors to compare their distribution using Matlab. How can I deal with the different length problem? Shall I divide each vector by its length?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
8
From what you've said, you have two sample sets:
$$x_n, n = 0 \ldots N-1$$ and $$y_m, m = 0 \ldots M-1$$
where $M \ne N$ and you want to compare the distributions of the underlying processes.
Do you know anything about the expected distribution of the measurements?
If they're Gaussian, then you can just calculate the sample mean and sample standard deviation (or variance) of $x_n$ and $y_m$ and if they're sufficiently close, then the measurements could be from the same underlying process.
If you can't assume Gaussianity (or any particular distribution), then it's possible to just do a histogram of each and compare the histograms. This page has a more detailed explanation.
Normalization
One way to normalize a time series is to use the $z$-normalization:
$$x'_n = \frac{x_n - \mu_x}{\sigma_x}$$
$$y'_m = \frac{y_m - \mu_y}{\sigma_y}$$
where $\mu_x$ and $\mu_y$ is the known mean of each time series and $\sigma_x$ and $\sigma_y$ are the respective standard deviations. The sample means and standard deviations can be used in lieu of the real ones.
This attempts to make $x'_n$ and $y'_m$ zero mean and unit variance.
-
$\begingroup$ But I think that the different size of the two vectors will affect the mean and variance of both distributions. In other words, if it is allowable to increase the size of the shorter vector such that both vectors have the same number of elements, then their mean and variance will be identical. With different size, the mean and variance are not identical. How can I avoid the effect of different size on the results? $\endgroup$ Commented Sep 14, 2021 at 14:10
-
1$\begingroup$ @FatmaDiab why do you think that the number of samples will change the expected sample mean and expected sample variance? If both sets of measurements are measuring from the same process, then they’ll both have (close to) the same sample mean and sample variance. If they don’t have the same sample mean and sample variance, then they’re drawn from different distributions. Can you explain a bit more about how you get $x_n$ and $y_m$? Perhaps that’ll make it clearer. $\endgroup$– Peter K. ♦Commented Sep 14, 2021 at 17:08
-
$\begingroup$ Thanks. It is clear now. Most papers use normalized score along the X axis. What type of normalization do they mean? How normalization is applied? On each vector separately or on both vectors simultaneously? If I have completely separable distributions before normalization, and the values of each vector are normalized to the range from zero to one, the two distribution will overlap, which is not correct. $\endgroup$ Commented Sep 14, 2021 at 21:35
-
$\begingroup$ @FatmaDiab One way to normalize added to answer. $\endgroup$– Peter K. ♦Commented Sep 14, 2021 at 23:40
-
1$\begingroup$ @HorseHair The link in that part of my answer gives some pointers as to when you would and when you wouldn't. Feel free to ask a new question if you want more details! $\endgroup$– Peter K. ♦Commented Sep 19, 2021 at 18:00