Can you give me some advices on what could be the best algorithm(s) to use for detection of fingertips/nails in image. First thing that crossed my mind was Viola - Jones. After rethinking I concluded that maybe it would be possible to use just Hugh transformation after applying edge detection. But I'd like some more advices. Also, as this is going to be a student's project with a purpose of learning, I'am not allowed to use OpenCV or similar frameworks. Below is typical image that will be processed. (note it's not top-down view). There is no need for thumb detection.
I would consider using neural network or SVM to fit the model. The difficulty with this approach is that you must collect a lot of data - both positive and negative examples. But you can generate a lot of artificial data(by scaling or rotating images you have already collected). After you collect data you can use "moving window" of few sizes to detect nails on images not present in training set. I don't know how much time you want to spend doing the project. But implementing efficient learning algorithm yourself is difficult. However there is already library for SVM which I would use for this purpose.
I have used Viola-Jones in OpenCV which was Haar. It is really powerful, based on wavelet, more than what I expected.
But is this the typical image you are targeting? What if the fingers out-stretched or straightened? or tilted?
You need to clearly define your domain.