52
$\begingroup$

I'm looking forward to enroll in an MSc in Signal and Image processing, or maybe Computer Vision (I have not decided yet), and this question emerged.

My concern is, since deep learning doesn't need feature extraction and almost no input pre-processing, is it killing image processing (or signal processing in general)?

I'm not an expert in deep learning, but it seems to work very well in recognition and classification tasks taking images directly instead of a feature vector like other techniques.

Is there any case in which a traditional feature extraction + classification approach would be better, making use of image processing techniques, or is this dying because of deep learning?

$\endgroup$
  • 3
    $\begingroup$ Reopening this because it has a high number of upvotes and the top-voted answer has a very high number of upvotes. $\endgroup$ – Peter K. Oct 28 '15 at 9:57
  • 1
    $\begingroup$ @LaurentDuval I think every answer was helpful and very interesting, but mainly yours and mathreadler ones (along with the discussion that came up) really clarified the topic. $\endgroup$ – Tony Jan 22 '17 at 15:37
  • 2
    $\begingroup$ I would like to make a crossroad in the ongoing discussion. Who said deep learning doesn't require feature extraction? In my own practical experience, we shouldn't train DNN for raw data. We have to do some feature extraction and also must possess some basic understanding of the image. Deep learning should be used with care, but its also a good idea. $\endgroup$ – arun raj Apr 10 '17 at 11:57

10 Answers 10

44
$\begingroup$

This post has been updated a lot. On the top, you can see link updates. Below, variations on the initial answer. For the short version: successes of convolutional neural networks and deep learning look like a sort of Galilean revolution. For a practical point of view, classical signal processing or computer vision are dead... provided that you have enough labeled data, care little about evident classification failures (deep flaws), have infinite energy to run tests without thinking about the carbon footprint, and don't bother rational explanations. For the others, this made us rethink about all what we did before: feature extraction, optimization (cf. my colleague J.-C. Pesquet work on Deep Neural Network Structures Solving Variational Inequalities), invariance, quantification, etc. And really interesting research is emerging from that, hopefully catching up with firmly grounded principles and similar performance.

Updated links:

We introduce natural adversarial examples -- real-world, unmodified, and naturally occurring examples that cause classifier accuracy to significantly degrade. We curate 7,500 natural adversarial examples and release them in an ImageNet classifier test set that we call ImageNet-A. This dataset serves as a new way to measure classifier robustness. Like l_p adversarial examples, ImageNet-A examples successfully transfer to unseen or black-box classifiers. For example, on ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%. Recovering this accuracy is not simple because ImageNet-A examples exploit deep flaws in current classifiers including their over-reliance on color, texture, and background cues. We observe that popular training techniques for improving robustness have little effect, but we show that some architectural changes can enhance robustness to natural adversarial examples. Future research is required to enable robust generalization to this hard ImageNet test set.

Deep learning references "stepping" on standard signal/image processing can be found at the bottom. Michael Elad just wrote Deep, Deep Trouble: Deep Learning’s Impact on Image Processing, Mathematics, and Humanity (SIAM News, 2017/05), excerpt:

Then neural networks suddenly came back, and with a vengeance.

This tribune is of interest, as it shows a shift from traditional "image processing", trying to model/understand the data, to a realm of correctness, without so much insight.

This domain is evolving quite fast. This does not mean it evolves in some intentional or constant direction. Neither right nor wrong. But this morning, I heard the following saying (or is it a joke?):

a bad algorithm with a huge set of data can do better than a smart algorithm with pauce data.

Here was my very short try: deep learning may provide state-of-the-art results, but one does not always understand why, and part of our scientist job remains on explaining why things work, what is the content of a piece of data, etc.

Deep learning requires (huge) well-tagged databases. Any time you do craftwork on single or singular images (i. e. without a huge database behind), especially in places unlikely to yield "free user-based tagged images" (in the complementary set of the set "funny cats playing games and faces"), you can stick to traditional image processing for a while, and for profit. A recent tweet summarizes that:

(lots of) labeled data (with no missing vars) requirement is a deal breaker (& unnecessary) for many domains

If they are being killed (which I doubt at a short term notice), they are not dead yet. So any skill you acquire in signal processing, image analysis, computer vision will help you in the future. This is for instance discussed in the blog post: Have We Forgotten about Geometry in Computer Vision? by Alex Kendall:

Deep learning has revolutionised computer vision. Today, there are not many problems where the best performing solution is not based on an end-to-end deep learning model. In particular, convolutional neural networks are popular as they tend to work fairly well out of the box. However, these models are largely big black-boxes. There are a lot of things we don’t understand about them.

A concrete example can be the following: a couple of very dark (eg surveillance) images from the same location, needing to evaluate if one of them contains a specific change that should be detected, is potentially a matter of traditional image processing, more than Deep Learning (as of today).

On the other side, as successful as Deep Learning is on a large scale, it can lead to misclassification of a small sets of data, which might be harmless "in average" for some applications. Two images that just slightly differ to the human eye could be classified differently via DL. Or random images could be set to a specific class. See for instance Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (Nguyen A, Yosinski J, Clune J. Proc. Computer Vision and Pattern Recognition 2015), or Does Deep Learning Have Deep Flaws?, on adversarial negatives:

The network may misclassify an image after the researchers applied a certain imperceptible perturbation. The perturbations are found by adjusting the pixel values to maximize the prediction error.

With all due respect to "Deep Learning", think about "mass production responding to a registered, known, mass-validable or expected behaviour" versus "singular piece of craft". None is better (yet) in a single index scale. Both may have to coexist for a while.

However, deep learning pervades many novel areas, as described in references below.

Luckily, some folks are trying to find mathematical rationale behind deep learning, an example of which are scattering networks or transforms proposed by Stéphane Mallat and co-authors, see ENS site for scattering. Harmonic analysis and non-linear operators, Lipschitz functions, translation/rotation invariance, better for the average signal processing person. See for instance Understanding Deep Convolutional Networks.

$\endgroup$
  • 1
    $\begingroup$ Augmenting insufficient training data using suitably modified copies helps deep learning to generalize. Lately, ways have been found around the need for complete supervised tagging: Unsupervised data augmentation automatically generates labels for the unlabeled portion of the training data in semi-supervised learning and uses that data for training. (Feel free to incorporate this or similar information in the answer.) $\endgroup$ – Olli Niemitalo Jul 20 at 9:37
  • 1
    $\begingroup$ If you know how to augment "consistently". OK on the classical datasets, still walking around on the scientific data I deal will (geology, chemistry) $\endgroup$ – Laurent Duval Jul 20 at 9:54
  • $\begingroup$ @Laurent, about what you said: "our scientist job remains on explaining why things work": sounds like data science is a valid career for somoeone considering working seriously on DSP. Are there any other names besides the typical "DSP engineer" title that you've heard of? $\endgroup$ – JFonseca Nov 1 at 22:03
21
$\begingroup$

First, there is nothing wrong with doing grad work in image processing or computer vision and using deep learning. Deep learning is not killing image processing and computer vision, it is merely the current hot research topic in those fields.

Second, deep learning is primarily used in object category recognition. But that is only one of many areas of computer vision. There are other areas, like object detection, tracking, 3D reconstruction, etc., many of which still rely on "hand-crafted" features.

$\endgroup$
  • 5
    $\begingroup$ Be careful: DNNs are very well capable of doing all of those that you mention: Object detection, tracking, 3D reconstruction, etc. That said, signal processing is an insight into the physical aspects of how signals are being manipulated, and why we should manipulate them in some way - and those (I believe) will make a come-back for explaining why adaptive algorithms like DNNs work. But make no mistake - DNNs are very well capable of basis transformations from the input, and all the way to the (differentiable) target objective. $\endgroup$ – Tarin Ziyaee Sep 14 '16 at 3:23
11
$\begingroup$

No Deep Learning isn't killing Image Processing. You need huge datasets and lots of computational resources to do deep learning. There are plenty of applications where it is desirable to be able to do image processing with less computational load and smaller memory footprints and without having access to huge databases. Some examples are mobile phones, tablets, mobile cameras, automobiles, quadcopters. Deep learning is very hyped right now as there exist some very impressive results to classification.

Classification is one problem out of many which Image Processing deals with so even if it were true that deep learning would solve all classification problems, there would be plenty of other types of Image Processing left to do. Noise reduction, image registration, motion calculations, morphing / blending, sharpening, optical corrections and transformations, calculating geometries, 3D estimation, 3D+time motion models, stereo vision, data compression and coding, segmentation, deblurring, motion stabilisation, computer graphics, all kinds of rendering.

$\endgroup$
  • $\begingroup$ Denoising, 3D estimation, etc, all those you mentioned are very able to be approximated and solved by DNNs of appropriate architecture, and appropriate data. $\endgroup$ – Tarin Ziyaee Sep 14 '16 at 5:27
  • 1
    $\begingroup$ Yes yes and you can do your weekly shopping in a Jaguar (but that's not why they are built). $\endgroup$ – mathreadler Sep 14 '16 at 9:47
  • 1
    $\begingroup$ Hehe, true - but that's different than saying that you cant shop using your jaguar. $\endgroup$ – Tarin Ziyaee Sep 15 '16 at 1:29
  • $\begingroup$ It is easy to impose useful constraints on an engineering problem which DNNs are rather crappy to handle. For example a constraint that the method used should not be biased towards a certain set of input data. Then DNNs will of course be disqualified by default as they all need training and therefore will be biased using training data. $\endgroup$ – mathreadler Sep 15 '16 at 11:03
  • $\begingroup$ That is true for any engineering tool: But that's not the point. The point is that all those tasks that you have mentioned above, can in fact very well be solved with DNNs. Yes, some are more recent developments, but it is mis-leading to say that they cant be solved with DNNs! That's all! $\endgroup$ – Tarin Ziyaee Sep 15 '16 at 16:40
11
$\begingroup$

Today we had a discussion with a friend of mine. It was a rainy day here in Munich, while a large portion of Europe was having a kind of sunny atmosphere. People were sharing photographs in social media, where they were in nice summer dresses, wandering around the seas. She was annoyed with this situation and turned to me and asked: "Could you write a software to block the pictures on social media, which involve such cute photos of summer, when the weather is this bad here?". I said, why not. All you need to do is to gather a huge set of summer images, and negative examples, feed it through a network, which does binary classification on the level of "Block" or "No-block". Train and tune the network. That's it.

Then, I turned to myself: Do I actually know how to write a simple algorithm to decide whether the weather is nice or not, without letting the machine to do the thinking for me? Barely... Maybe... For the curious reader, here is some features that you might want to design, if you would try to go for it :

Two-Class Weather Classification, Cewu Lu§ Di Lin, Jiaya Jia, Chi-Keung Tang, CVPR 2014

Obviously, I wouldn't even care about this CVPR publication nowadays and just go deep. So, as much as I like the deep learning for its robust performance in many scenarios, I also use it cautiously. Even if it wouldn't kill my knowledge of image processing, it tends to decrease the domain expertise I require. Intellectually, this is not very elegant.

As soon as the individual decides to keep him/herself on track and benefits from both worlds, (s)he'll be on the safe side.

$\endgroup$
7
$\begingroup$

The short answer is, No. DL can recognize a mug in a photo, but this doesn't kill signal processing in anyway. That said, your question is quite relevant in these troubled days. There is a nice panel discussion on the subject, featuring Stephane Mallat, etc., here.

$\endgroup$
5
$\begingroup$

Data engineering is still used in machine learning to preprocess and select the data fed to DNNs to improve their learning time and their evaluation efficiency. Image processing (the stuff between the camera sensor and the RGB/etc. bitmaps fed to the DNNs), a form of data engineering, is still needed.

$\endgroup$
4
$\begingroup$

A thorough understanding of signal processing (along with linear algebra, vector calculus, mathematical statistics etc.) is imo indispensable for non-trivial work in the field of deep learning, especially in computer vision.

Some of the high impact papers in deep learning (now that most of the low hanging fruit have been picked) evince a good understanding of signal processing concepts.

A few motivational concepts:

  • Dilated convolutions: check out this blogpost. One of the first equations there would be bread-and-butter for a person well-grounded (hah) in signal processing concepts. It is also closely related to the a trous algorithm found in classical wavelet signal processing.
  • Transposed convolutional layers / Deconv layers. Again, basic signal processing concepts.
  • Shaping of conv filters - good idea of operator norms and contraction mapping is needed. This is usually found in either a grad EE course in Signal Theory or Control Systems, or in the Math courses on Analysis (Real or functional).
  • Adversarial examples: one of the first papers to investigate this ("Intriguing properties of ...") formalized this in terms of perturbations and used Lipschitz constants of the various layers and non-linearities in a neural network to upper-bound the sensitivity to such perturbations. Agreed, the analysis was very preliminary, but again I believe it proves the point that making non-trivial progress in anything, deep learning included, requires non-trivial understanding of theory.

The list goes on. So, even if you end up working in computer vision and applying deep learning to your problems, the signal processing background will make things very easy to grasp for you.

$\endgroup$
  • 1
    $\begingroup$ Yes. Any shortcut taken to not have to learn what to feed to the network will have to be learned the hard way by worse performance. $\endgroup$ – mathreadler Oct 1 '17 at 9:09
4
$\begingroup$

I really don't do much image processing but I worked for an organization (US Navy) that did and funded research in signal classification the last time Neural Nets were a hot topic, the mid to late 80's. I had to sit through a large number of essentially marketing stuff. The arguments were along the lines of:

  • It's Neural, like your brain and since it outperformed a linear classifier, it beats statistical techniques. I actually know some folks who had their papers rejected because they used statistics to evaluate performance.
  • Neural Nets are inductive, they can classify stuff correctly if none or few examples were in their training sets.
  • DARPA is funding work, and we all know that everything DARPA does is a winner.(Google wasn't around yet)
  • Isn't performance wonderful, don't need a confusion matrix, don't need class priors, I can just tell you what my probability of error is. Don't need bounds, I'l just do the hold-one-out and retrain shuffle.
  • Pick some features and go for it, its a black box, scaling, data alignment, clutter rejection, bad labels, occurrence of multiple classes, not my problem.
  • The Fog of Math, Boltzmann Machines
  • Let's throw in an SVD and maybe a fractal dimension thingy.
  • The supervised/unsupervised bait and switch, I'll find all your hidden patterns. Isn't this associative memory thing profound?

It took Bishop's book to tamp down my cynicism.

In more than a few applications, the optimal signal processing algorithm will require an exhaustive enumeration search over a large parameter space which quickly becomes intractable. A big server farm can increase that achievable search space but at some point, you need to find a heuristic. DL seems to be able to find some of those heuristics but it doesn’t solve the underlying NP hard optimization.

$\endgroup$
  • $\begingroup$ You are completely right in damping down cynicism as it seems to put many people down. I sometimes wish I learned that earlier in life. $\endgroup$ – mathreadler Oct 26 '17 at 21:34
3
$\begingroup$

My perspective from university was that many signal processing people were a bit hostile toward ML, I suspect because they felt threatened that it was encroaching on their domain. But recently there's been a lot of research into benefits of complex valued deep neural networks, which may suggest that the golden ticket is really a solid understanding of both disciplines.

$\endgroup$
  • 1
    $\begingroup$ Yep. Signal processing is very closely related to machine learning. A solid understanding of signal processing helps understanding how to build and to use ML algorithms and what kind of data is (un)suitable to feed them with. $\endgroup$ – mathreadler Jul 1 '18 at 10:49
2
$\begingroup$

Well, yes. In the same way that the development in higher level programming languages like C++ and Python 'killed' assembly programming. That does not mean it is irrelevant to learn assembly when you enroll in a CS course however. It provides great insight in how the computer works, what goes on behind the scenes of higher level languages, what the basic principles of computer language are, etc. But nobody in his right mind would now program a desktop app in assembly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.