4
$\begingroup$

I'm planning to acquire between 50k and 200k image per day with a 50MPixels (or 68MPixels or 130MPixels) sensor; I'll be acquiring the raw data (10 or 12 or 14 bits) from the sensor through SLVS-EC and create a raw file of my own design. The raw bitrate from the sensor may go up to 75.2 Gbps.

I may have to store 50k-250k images per day (eg., 17.5TB if 250k images are 70MB-50MPixels each). I need to keep high quality images (in particular, colors must remain accurate and textures fully detailed, hence the lossless or only light loss and nothing below 10 bits per channel), and a flexibility in edition (hence the raw).

Also images will share a lot, since I may have 2-24Hz framerate at capture; also a first processing will drop (delete) between 10% and 50% of images, so a keyframe based compression may not be suitable.

Since I need to keep the storage cost as low as possible without doing too hard compression (maybe go below 30-50MB per raw image). I'm planning to allow compression within this raw file, this compression can be lossless or lightly lossy. I'm thinking about wavelet and auto learnt dictionaries (patchs and sparse coding) for the compression, but this is not a requirement.

I will not release any sdk or raw image, so there are no need or requirement on the standard and adoption side. I'll very likely use an FPGA for signal processing (up to 75.2Gbps from the sensor), since I need very high IO and fast signal processing, and the whole package will be embedded, and as compact as possible and reasonnably light (say less than 1-2kg).

About the images, it will be natural environment with natural day light; it may include shadows and sky with sun, and hence high dynamic, but also rich (high frequency) textures which must be preserved. So likely I won't add further denoising, but I want to keep the fexibility with color processing: in particular the ability to change the signal amplification and the white/black balance.

Do you have thoughts and pieces of advice about the compression strategy for this raw format ? In particular do you think video compression algorithms (eg., HEVC) could be adapted to raw bayered data ?

$\endgroup$
12
  • $\begingroup$ none of this suggests an FPGA is the right approach! You need to store data on a relatively benign rate on mass storage – sounds like a job for PC-style general purpose hardware, i.e., desktop/workstation/server CPUs. So, if you need the FPGA for anything else, OK, but trying to do the processing on the FPGA will both be much more complicated, and much more expensive, than doing it on a CPU. $\endgroup$ Feb 8, 2022 at 20:24
  • $\begingroup$ There might be sense, under some very specific applications, to build an FPGA-based accelerator for some parts of the signal processing. For what you've written so far, this isn't necessary nor would it be productive. So, you'll have to tell us, really, what your system is for, what the things limiting it are, why you need to use an FPGA (because you'll have some PC-sized processing in this system, anyways, in all devices that store these amounts of data), and so on. Please clarify! Your question isn't very well-defined without, I'm afraid. $\endgroup$ Feb 8, 2022 at 20:32
  • $\begingroup$ @MarcusMüller Thanks for all the comments. I edited in order to improve clarity. If your read the OP again, you will find that I did not used "need to use FPGA", but instead "likely", the same way I used "keyframes may not be suitable". $\endgroup$
    – Soleil
    Feb 9, 2022 at 0:10
  • $\begingroup$ Doing "something" in MATLAB/Python may cost X. If so, I think that doing the same thing in clean, fast, tested C is at least (licking my finger) 10X. Doing the same in native fpga logic might well be another 10X. If the application calls for the speed and/or efficiency afforded by low level tools, then you need to invest in the knowledge/coworkers that is required. In many cases, if you can buy yourself out of the problem by running an expensive fan-cooled PC, that may allow you to focus on other, more pressing tasks. $\endgroup$
    – Knut Inge
    Feb 9, 2022 at 7:19
  • 1
    $\begingroup$ There are raw codecs out there. Apple have something called ProRes RAW, REDCODE can compress by 18:1. You will probably find that lossless intra compression can do something like 2:1 compression on a varied set of input, that specialized «raw» codecs have moderate compression, while you need something like h26x with inter compression to have 100:1 compression with decent quality. So it depends on how much of a pain storage cost it to you, what you need to retain in the images, and how much time/competence you have to tailor a solution. $\endgroup$
    – Knut Inge
    Feb 9, 2022 at 13:35

1 Answer 1

2
$\begingroup$

Raw files are (ideally) the raw readout of a sensor. Suitable for research, or if you want to eek out all possible information from a sensor using fancy offline processing. Now or in 10 years. In some cases, you might not need all of the information contained in a full raw image, but be satisfied with having maksimum exposure freedom - ie to avoid tonemapping/tonecurves baked into something like a jpeg preprocessing.

Do you specifically need to store it in Bayer format, or could you do debayer and use some off the shelf YCbCr 4:2:0 compression? What kind of compute platform do you have between sensor and storage (a PC?)

If file size is a major concern, something like x264 with >8 bits and high-ish bitrate is going to be hard to beat with home-grown tools in terms of quality per bit or quality per cycle unless you have very specific requirements or a lot of skill and time.

Edit: Responding to some of the comments below. I would borrow a nice camera, take two snapshots of the scene in question, read the raw files using dcraw or some similar tool and import into matlab/python.

Then you can play with debayer, fixed whitebalance (?), whitepoint and blackpoint and gamma (I think that h26x tends to be limited to 10 bits, but note that this is usually nonlinear quantization - more resolution in the blacks where it matters more). Finally, do a 3x3 matrix to a pseudo YCbCr format, save and pass it to a lossy encoder. Observe the file size of the first frame (intra) and the second (inter). That tells you a lot about how compressible the stream will be.

Then check the output file, carefully reversing the steps above. Check if quality is sufficient for your needs. Be prepared to do some fiddling until the stars align.

$\endgroup$
10
  • $\begingroup$ Did you mean x265 ? I'll have an FPGA for signal processing (OP updated). $\endgroup$
    – Soleil
    Feb 8, 2022 at 18:03
  • $\begingroup$ If you are filming something like a security camera, vanilla compression w/inter prediction may do well even with every second frame deleted. I dont think x264 or x265 are available as fpga code, but perhaps the fpga manufacturer provides libraries with standard video compression? Or are you running pure software on some hard cpu block? Doing cutting edge video compression in native fpga sounds like real work… $\endgroup$
    – Knut Inge
    Feb 8, 2022 at 18:23
  • $\begingroup$ Well, I also prefer to avoid video compression with keyframes, but since there will be a lot in common, I may think of a sparse dictionary common to several images, so I avoid keyframes and I can drop the 10-50% of images easily, and still have a wavelet based comrpession. x264/x265 are subsequently rather opted out. The problems is that it may be too much "CPU bound" since I need to keep a good write speed. $\endgroup$
    – Soleil
    Feb 8, 2022 at 18:28
  • $\begingroup$ where does the need for wavelets arise from? 30 years of compression research suggest wavelets are not the tool of choice here. Also, note you're coming up with new requirements in every comment. This suggests you need to fundamentally rewrite your question to actually explain your constraints, rather than letting them negate an answer after people have spent time on answering! I, for example, would have an answer to complement Knut's answer, but I will not spend half an hour writing it down unless I understand why you're having these rather surprising constraints – $\endgroup$ Feb 8, 2022 at 20:28
  • 1
    $\begingroup$ If you debayer, you loose some freedom to do fancy debayer, and exotic joint denoise/deconvolution/debayer algorithms. You can still keep the color representation native for full flexibility in color corr/white balance. You can still keep a linear tone curve meaning that exposure correction and highlight recovery can do its thing. Note that having a linear gamma may reduce compression efficiency of jpeg/h264 somewhat. $\endgroup$
    – Knut Inge
    Feb 9, 2022 at 5:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.