The kernels used by a ConvNet are nothing but neural weights. You can think of them as a multilayer perceptron with some connections cut off and some weights restricted to be equal (weight sharing).
With this in mind, we must take into account that the kernels (or filters, in this context) are learned, so they depend exclusively on the type of inputs and the desired outputs. Each learning process is unique for a set of inputs/outputs, and so there is no way to know a priori what the filters will turn out to be. They are, as you state in your question, application-dependant.
They can be interpreted, though. Take for example a simple case, where we are trying to classify images that consist of horizontal lines only, and others of vertical lines only. If you do this, then the kernels will indeed look like that: vertical and horizontal lines. That's because the filters have learned common patterns to look for on the inputs, and they have acquired that form in order to be able to match those patterns and maximize their correlation to them.
In Figure 3 at Krizhevsky et al., there are some kernels for you to see. Kernels looking for horizontal lines will consist of horizontal lines. Kernels looking for vertical lines will consist of vertical lines. And so on.
To see this more clearly, you can train a CNN and take one of the kernels learned and filter any image with it to see what happens.
I think that a paragraph from this website can be of relevance:
So what can we conclude from these feature maps? It's clear there is spatial structure here beyond what we'd expect at random: many of the features have clear sub-regions of light and dark. That shows our network really is learning things related to the spatial structure. However, beyond that, it's difficult to see what these feature detectors are learning. Certainly, we're not learning (say) the Gabor filters which have been used in many traditional approaches to image recognition. In fact, there's now a lot of work on better understanding the features learnt by convolutional networks. If you're interested in following up on that work, I suggest starting with the paper Visualizing and Understanding Convolutional Networks by Matthew Zeiler and Rob Fergus (2013).
In the paper mentioned in the previous quote, there are lots of examples of learned kernels, related to the images that were used to train the network. I think that paper can give you some useful insight.