I have a project I'm working on where I'm trying to efficiently compare sequences of binary data to determine how similar they are. Basically, a sequence might look like this: 1001010010100011110...
I spent a while on my own trying to come up with a way to do it because it's fun for me to work through problems like this on my own, and actually made some progress. Unfortunately, everything I came up with never quite worked; a solution that accounts for substitutions (bits that flip) is easy, but what keeps getting me is handling bits that are inserted or deleted, i.e.: 100101 -> 1000101
Trying to find a solution online keeps leading me here, and I keep coming across the term cross-correlation. I also see lots of interesting links on the wikipedia page for Time Series for calculating similarities. Unfortunately, at the moment, a lot of the math is unclear to me, and I can't efficiently go through and distinguish what the appropriate technique might be for my situation.
So I have two different approaches to my problem that I'm interested in learning about:
1) What method would you recommend for calculating the similarity between two bit sequences (typically 50 to 150 bits in length) that is robust to substitutions, insertions, and deletions?
2) I'm not sure this is possible, but is there some kind of distance metric(s) I can calculate independently for each signal that will allow me to group them together with actually doing direct pairwise comparisons?
For 1, I feel like cross-correlation might be the right approach, but I haven't gotten past just plugging sequences in R and seeing how it works for identifying lag. I know edit distance will work, but ultimately will want something more efficient. Eventually, I will be trying to group billions of these together.