This is not a simple problem. Almost certainly any "home-brewed" system you create looking for "frequency patterns" is going to have horrible performance, unless it develops complexity comparable to full-fledged speech recognizer. The reason for this is that recognizing speech, while it seems like a simple task to our ears, is in fact quite complex, and requires large amounts of context to have good accuracy. What I'm trying to say is that if you just try to recognize the phonemes from the audio locally, you're going to get gibberish. You need to take into account the neighboring phonemes, the vocabulary of the language being spoken, and the large scale structure and content of the surrounding sentences in order to get anywhere near human level performance.
At the end of the day if you try to write something which "recognizes frequency patterns of speech" from scratch, you're going to be reinventing the wheel, and a pretty dang big wheel at that.
For this reason, the best thing for you to do would be to take an existing speech recognition system, and modify its output to suit your needs. I would suggest using CMU Sphinx to decode the audio into an N-Best list, and then search this list for the text you are looking for. CMU Sphinx has good recognition performance and is highly configurable.