We have the need to save signal samples measured with an arbitary sampling rate. The samples are usually about one second in duration. Currently we save them in a text file format which consumes a lot of disk space and is very slow. Also this type of format doesn't suit for streaming applications.
I have been searching for a binary file format for this kind of data, but I'm not sure what factors I need to consider when choosing a format.
The sampling rate of our data varies from 1kHz to 152kHz and currently 16 bits would be enough to represent the data (varies roughly from -1000 to 1000). This is vibration data measured from different sensors if it makes a difference. Also we would need to save pre-calculated FFT's in the same format. Is this possible?
Just to point out, I have no education in signal processing. I have the task to implement the system to be used in the analysis.
Currently the best option seems to be WavPack, which is basically a compressed WAV file and supports sampling rates from 1 to 4.3GHz and the compression is lossless.
WavPack is released in 1998 so I'm thinking are there any newer or more efficient formats available today?