4
$\begingroup$

Recently, convolutional neural network based, deep architectures (DNN) such as AlexNet and VGGnet have been very successful in image classification challenges (e.g. ImageNet) and action recognition/video classification tasks. They surpassed hand-crafted feature-based approaches by a large margin in various cases, and thus became the new state-of-the-art approach for many computer vision problems, including but not limited to classification, object detection, and action recognition.

In the state-of-the-art DNN papers, the focus is mostly on achieving a better classification or detection accuracy, whereas the complexity of the model and computational issues stay rather in the background. This made me think whether these DNN-based approaches would be suitable in future real-time action recognition applications (e.g. video surveillance, pedestrian detection etc.) where the test time speed is of utmost importance. It is very important, for example, whether an algorithm can classify the captured video in real-time and respond very quickly to the classification decision to take some pre-determined action.

Are there any DNN-based action recognition algorithms published so far that can satisfy these criteria about real-time operation (possibly even at the expense of classification accuracy)?

I believe for many practical applications, achieving real-time or even faster-than-real-time performance is and will be much more important than providing marginally more accurate classification results compared to the previous state-of-the-art-.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I always thought that the training was the time-consuming part of any classifier, not the actual classification. The only (quickly available) reference I can find refers only to training, not classification. Do you have any references as to why DNNs are slow in the actual classification part? $\endgroup$
    – Peter K. ♦
    Mar 31, 2016 at 18:24
  • 1
    $\begingroup$ Training time is much more consuming that the test time (or inference e.g. classification) due to the back-propagation where the weights of the network are updated using gradient descent. Recently, I came across an interesting work which uses binary weights and activation and thus, reduces required memory size and access by replacing most arithmetic operations with bit-wise operations. Seeing similar works have led me to think that standard deep neural networks perhaps do not meet the latency requirements for various real-time applications. $\endgroup$
    – chronosynclastic
    Apr 1, 2016 at 9:20
  • $\begingroup$ I wonder if you can get a rough estimate of computation time based on the number and size of the convolutions. $\endgroup$
    – geometrikal
    Apr 1, 2016 at 22:22
  • $\begingroup$ I suppose to update an answer, there are processors specifically designed for efficient convolution calculations. Real-time deep network testing is an existing technology. (See en.wikipedia.org/wiki/AI_accelerator) $\endgroup$
    – adfriedman
    Jul 8, 2017 at 8:54

2 Answers 2

Reset to default
7
$\begingroup$

Nvidia seems to have published some white papers comparing DNN inference performance between high-powered CPUs and (of course) Nvidia GPUs. (one example)

Ballpark seems to be that some systems can meet or exceed typical video frame rate thru-put for some class of DNN classification tasks. Whether those image sizes and/or DNN architectures and hyperparameters meets your needs is one question, and whether desktop/server-class GPUs fit your desired power envelope is another question.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Very interesting white paper! The provided results show indeed that even for a relatively complex model such as GoogLeNet, an SoC (Tegra X1) GPU (Maxwell-based 256 core) can achieve real-time classification performance (assuming a frame rate of 30 fps). State-of-the-art GPUs (e.g. Titan X) can achieve much higher. Of course, as you mentioned, the results are highly dependent on other factors such as video resolution, quality as well as DNN architecture and various hyper-parameters but I think the results look very promising for real-time processing on end-user devices such as smartphones. $\endgroup$
    – chronosynclastic
    Apr 1, 2016 at 9:30
2
$\begingroup$

Here is a newly published paper and video example:

https://www.youtube.com/watch?v=w2iV8gt5cd4

http://arxiv.org/abs/1411.4389

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi. How can you cite that the youtube link content is authentic ? $\endgroup$
    – Fat32
    Jun 19, 2016 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.