Here's something that should work:
1) Find the center of mass of all your points.
2) Sweep a radius around this center.
3) Measure a few metrics along each radius, generating a set of "periodic signals"
4) Take the DFT of the signals to determine magnification and rotation (as phase shifts).
Off the top of my head and similar to this:
Auto Detection of ...
A Hough transform can certainly work. However, with position, scale, and orientation, you would have to use a 5-D Hough transform (2 DOF for position, 1 for scale, 1 for orientation), which might be computationally quite demanding.
As an alternative, you might want to look at SIFT or one of its extensions like SURF. It sounds like they can be adapted to do ...