Selecting a Digital Signal Processor for a Specific Image Sensor

Question

I am working on developing a camera and I am trying to select hardware for the camera. I believe that I may need a digital signal processor to compress image data before it is transmitted to a computer or other device (MCU) . What I am trying to figure out is how to estimate the processing power needed given a specific DSP. For example I was considering the OV5670 Image Sensor. This sensor is capable of transmitting 5 megapixel image data at 30 fps. If each pixel is stored as a 10 bit value that means in total 1.5Gbps would be being transmitted to a digital signal processor of my choice. The reason I am doing compression is that my ideally the camera would record data and then upload it to a cloud based server. I think it would be ideal then for compression to take place before uploading the data to reduce bandwidth usage.

From here I am uncertain how to proceed. For example I examined the ADSP-BF529BCPZ DSP chip. This chip can execute instructions at a rate of 400 MHz. In the data specs it is specified that the processing core of this chip contains

"Blackfin processor core contains two 16-bit multipliers, two 40-bit accumulators, two 40-bit ALUs, four video ALUs, and a 40-bit shifter."

Now each MAC can do a 16 bit multiplication and accumulate the result in the 40 bit shifter. There are 2 such MACs. If I were trying to compress the data by taking a DFT and then selecting the largest coefficients then I could say to calculate a single coefficient in the frequency domain I would need to multiply each pixel by a value and then sum it. Between the 2 MACs this would take 2.5 million cycles for one coefficient of a single frame.

Based on the above calculations it seems unlikely that the selected chip would be able to handle the compression as I need it to. But at the same time I know algorithms exists such as the FFT which can drastically boost the speed of the DFT.

What I am looking for is either confirmation that my current intuition into selecting a DSP chip is correct or alternatively that I am over/under-estimating the capabilities of DSP chips. In the later case, if possible please suggest other methods for predicting the required processing power from a DSP for a specific image compression application.

Thank You in advance, if any additional information is needed please let me know. Also feel free to suggest other solutions to my problem. For example, if you believe that I do not need a DSP and could instead achieve my goals through either a MCU or an FPGA then feel free to make such a suggestion.

edit: Also if anyone looks at the datasheets for the above listed components you will see that the image sensor transfers data via MIPI-CSI2 protocol. Which the DSP chip does not support. I am aware of that shortcoming but it is not the main driver of my current question.

Answer 1

Good approach to first do a rough calculation of the bandwidths you need.

Couple of remarks on that:

• You forgot a factor of 3, that camera has three "color pixels" per image pixel
• Your use case screams "I should be using a commercial off-the-shelf USB camera"; don't engineer something very complex if it doesn't have a value proposition over what you can buy. If the value proposition is "I can learn how to work with cameras, and DSP", I'd frankly start smaller, and learn how to work with camera images on a PC in software before moving out part after part to an embedded processing platform.
• You try to downplay the camera interfacing, but it's really a critical part here: in need to interface digital buses for which their processors have no dedicated interface, engineers turn to FPGAs. You'll need one, just as the "glue" between your camera
• Your approach of using an MCU betrays that you're not overly familiar with what these can and can't do. Wrong order of magnitude: for this kind of problem you need what is usually sold as "application processor", with plenty RAM, and things like interfaces for network hardware and the opportunity to run an OS reasonably fast.
• You picked an oldish low-power DSP core. That 400 MHz thing can do way less than modern application processor CPUs can do. For example, every SSE-enabled x86 CPU can do two multiplications at once, just like that DSP core, but they tend to run much faster.
• You're planning to stream something between 1080p and 4k video .. to the internet. You really need a fast internet uplink to do that. You might want to download a 4k video snippet and look at the amounts of bits per second that needs.

So, all in all:

• You need an application processor, which take your MCU completely out of business (you will probably still need an MCU, to do things like turning on power to the different power rails of your other components in the right order, but not at all involved in handling the camera data)
• You need a device with a MIPI interface. If you're able to buy in big numbers, you'd typically get a camera or smart phone System-on-Chip that has a CPU (see above) which has such an interface. In your case, it's going to be an FPGA, or a MIPI-to-USB3 bridge (cypress sells these). In any way you do it, getting the interface right isn't trivial.
• I don't really see how a DSP core fits in here in practice: for the data / camera handling, your application CPU does the most. For the video encoding, you'd typically use a dedicated encoding accelerator that does the computationally hardest part of it.

All in all, this sounds like you'd start with a small x86 mainboard (there's way smaller form factors than µATX, but honestly, µATX isn't so large for prototyping), and get a camera that you can attach to that - and that would usually simply be a high-quality webcam or DSLR.

For highly integrated systems, going the FPGA route is very attractive, and you can do a lot of computation by writing the right hardware description - problem is, that takes a lot of experience. For even higher integration, you'll find ASICs that are both pretty camera-specific and use-case-specific that take on the task of handling everything from camera to radio interface. In fact, DJI has bought a semiconductor manufacturer just so that they have that kind knowledge in-house, although they are by no means a semiconductor company – they sell video drones.

Honestly, I'd call the FPGA approach "unrealistic within the next year", given your current understanding of the matter, sorry. The ASIC route is impossible, since production of a single high-speed wafer costs millions, and you don't have a silicon design team.