Unfortunately, I dont have access to a MATLAB image processing toolbox. I've coded a solution in python, showing the idea of how to solve it:
import skimage
import skimage.io
import scipy.ndimage as ndi
import skimage.morphology as m
import skimage.measure as measure
plt.figure(figsize=(8, 12))
I = skimage.io.imread("https://i.stack.imgur.com/eCVzl.jpg")
plt.imshow(I[:,:,2])
bw = I[50:250,100:300,1] < 100 # Import and binary image. Cut the region of interest (e.g. remove the white border)
plt.subplot(321)
plt.imshow(bw, interpolation='none')
plt.subplot(322)
plt.title('Distance transform of input image')
R, D = m.medial_axis(bw, return_distance=True) # Calculate the distance transform (Matlab: bwdist)
plt.imshow(D, interpolation='none')
plt.subplot(323)
plt.title('Convex hull of input image')
C = m.convex_hull_image(bw == 0) # Calculate the convex hull of the pixels (i.e. the are where the maxima can be)
plt.imshow(C)
masked = C * D # zero-out all distances that are not within the convex hull
plt.subplot(324)
plt.imshow(masked)
plt.title('Masked distances according to convex hull')
M = measure.moments(C.astype(np.uint8), order=2);
cy, cx = M[0, 1] / M[0, 0], M[1, 0] / M[0, 0] # Calculate the convex hull center of mass
x, y = np.meshgrid(np.arange(C.shape[0])-cx, np.arange(C.shape[1])-cy);
weights = np.exp(-0.003*(x*x+y*y)) # calculate a weighting mask
plt.subplot(325)
plt.imshow(weights)
plt.title('Distance weights')
masked_weighted = masked * weights
plt.subplot(326)
max_pos = np.unravel_index(masked_weighted.argmax(), masked.shape) # detect maximum pixel and its 2d-coordinates
plt.imshow(masked_weighted)
plt.plot(max_pos[1], max_pos[0], 'kx', mew=2)
plt.title('Weigthed distances with detected center')
plt.subplot(321)
plt.plot(max_pos[1], max_pos[0], 'kx', mew=2)
plt.title('input image with detected center')
plt.tight_layout()
The idea is based on an operation called "Distance Transform" and that the center of a circle is the point that has the largest distance to its boundary. The program assumes that the center of the circle is roughly in the middle of area spanned by the edges. Furthermore, it assumes that the maximum distance pixel corresponds to the circle center (i.e. other edges are closer together). So, in rough steps:
- calculate distance transform of Image
- calculate convex hull of the edge pixels
- caluclate center of the convex hull to detect the rough center
- weight the obtained distances from 1) by the distance from the estimated center
- use the strongest pixel as the center of the circle.
The code sohuld be easy to translate into Matlab, esp. as I have plotted all intermediate images.