Here is nice resource with 'guidelines for use': homepages.inf.ed.ac.uk/rbf/HIPR2/hough.htm (sorry if the link rots)

The Hough transformation can take some fiddling to get right for your use case. The two problems here, it seems, are that it is connecting disparate lines just because they line up and that you are only getting the most significant of these. As you simplified your problem by preprocessing the image, you can lower your threshold a lot to bring in more lines. Also, perhaps a sliding window based approach would give better detail. We shouldn't have a long conversation here though, if the Hough transform brings up questions you'll find lots of folks willing to answer specifics on here. Cheers.

PS: you may be able to clean up your binary images a bit further using binary morphology techniques.