I want to kick up a discussion on how people set algorithmic parameters when no validation against groundtruth is possible (maybe because groundtruth just cannot be obtained or is very hard/tedious to obtain).
I have read numerous papers and implemented the underlying algorithms wherein --- a set of parameters are said to have been set "empirically" --- and often i found that these are the ones that affect the generality of the algorithm (even though the theory underlying the method is elegant, enticing and sound).
I would appreciate it if you could share your thoughts. And, there is no right or wrong answer for this question. I just want to know, how everybody else deals with this.
I am a computer scientist working in the areas of image analysis, computer vision, and machine learning and this question has been on the back of my mind for a while as i have faced this dilemma time and again whenever i design a new algorithm and i found myself spending a considerable amount of time tuning the parameters.
Also, I think, my question here is more general to any area where in computational algorithms are heavily involved, and i want to invite the thoughts of people from all concerned areas.
I wanted to give you some concrete example, just so it helps you think:
--- Take the case of feature detection (lets say circular blobs or salient points). You run some filters (needs parameters) at different scales (scale parameters) and probably threshold the response (threshold parameter). Its usually not possible to get a groundtruth to validate against and thereby automatically tune your parameters in such scenarios.
--- Take any computational framework which involves a lot of signal processing components. There are always parameters to tune and usually there is no groundtruth and when you tune them subjectively on a small random subset of your dataset you will someday encounter the case to which it doesnt generalize.
This parameter devil is more troublesome when you are setting parameters for some intermediate steps in your algorithm.
And I often found, it is not possible to cast the problem of finding good values for these parameters as an optimization problem with an objective function of which you can take a derivative and thereby use standard optimization algorithms to find good values.
Also, in many scenarios exposing these parameters to an end-user is not an option, as we often develop applications/software for non-computational end-users (lets say biologists, doctors) and they usually go clueless when you ask them to tune it unless its very intuitive (like approx object size).
Please share your thoughts.