I believe that you don't really want a signal processing (mathematical) solution to
your problem, instead the following programming approach could be a helpful too.
Assuming that the speakers in the A2 data set, do not alter the word ordering, then
you know that each audio file in the second set has exactly the same number of words spoken by different people.
Then you should look for a decent speech-recognition (speech to text) library that has at least the capability of isolating spoken words from each other, and indicating the beginnig and edning timings of each isolated word. You don't need to know what is being said, all you need is their isolation from each other.
Then having the output of such an isolated set, you will map one by one every word in the reference audio to the corersponding isolated word in the test set.
Depending on the isolation algorithm sucess rate, you mey end in wrong results though. No algorithm can guarantee error free isolation anyways.
If you need a more robust approach, of course you can look for full recognition library with the spoken word as a text output. Then you will now better.