12
$\begingroup$

I am happy to accept suggestions either in R or Matlab, but the code I present below is R-only.

The audio file attached below is a short piece of conversation between two people. My goal is to distort their speech so that the emotional content will becomes unrecognisable. The difficulty is that I need some parametric space for this distortion lets say from 1 to 5, where 1 is 'highly recognisable emotion' and 5 is 'non-recognisable emotion'. There are three ways I thought I can use to achieve that with R.

Download 'happy' audio wave from here.

Download 'angry' audio wave from here.

The first approach was to decrease the overall intelligibility by introducing noise. This solution is presented below (thanks to @carl-witthoft for his suggestions). This will decrease both intelligibility and emotional content of the speech, but it's very 'dirty' approach - it's hard to make it right to get the parametric space, because the only aspect you can control there is an amplitude (volume) of noise.

require(seewave)
require(tuneR)
require(signal)
h <- readWave("happy.wav")
h <- cutw(h.norm,f=44100,from=0,to=2)#cut down to 2 sec
n <- noisew(d=2,f=44100)#create 2-second white noise
h.n <- h + n #combine audio wave with noise
oscillo(h.n,f=44100)#visualize wave with noise(black)
par(new=T)
oscillo(h,f=44100,colwave=2)#visualize original wave(red)

enter image description here

The second approach would be to somehow adjust the noise, to distort the speech only in the specific frequency bands. I thought I could do it by extracting amplitude envelope from the original audio wave, generate noise from this envelope and then re-apply the noise to the audio wave. The code below shows how to do that. It does something different than noise itself, makes the sound cracking, but it goes back to the same point - that I am only able to change the amplitude of noise here.

n.env <- setenv(n, h,f=44100)#set envelope of noise 'n'
h.n.env <- h + n.env #combine audio wave with 'envelope noise'
par(mfrow=c(1,2))
spectro(h,f=44100,flim=c(0,10),scale=F)#spectrogram of normal wave (left)
spectro(h.n.env,f=44100,flim=c(0,10),scale=F,flab="")#spectrogram of wave with 'envelope noise' (right)

enter image description here

The final approach might be the key to solve this, but it's quite tricky. I found this method in report paper published in Science by Shannon et al. (1996). They used quite tricky pattern of spectral reduction, to achieve something that probably sounds quite robotic. But at the same time, from the description, I assume they might have found the solution that could answer my problem. The important information is in the second paragraph in the text and note number 7 in References and Notes - the whole method is described there. My attempts to replicate it so far have been unsuccessful but below is the code I managed to find, together with my interpretation of how the procedure should be done. I think that almost all the puzzles are there, but I can't somehow get the whole picture yet.

###signal was passed through preemphasis filter to whiten the spectrum 
#low-pass below 1200Hz, -6 dB per octave
h.f <- ffilter(h,to=1200)#low-pass filter up to 1200 Hz (but -6dB?)

###then signal was split into frequency bands (third-order elliptical IIR filters)
#adjacent filters overlapped at the point at which the output from each filter 
#was 15dB down from the level in the pass-band
#I have just a bunch of options I've found in 'signal'
ellip()#generate an Elliptic or Cauer filter
decimate()#downsample a signal by a factor, using an FIR or IIR filter
FilterOfOrder()#IIR filter specifications, including order, frequency cutoff, type...
cutspec()#This function can be used to cut a specific part of a frequency spectrum

###amplitude envelope was extracted from each band by half-wave rectification 
#and low-pass  filtering
###low-pass filters (elliptical IIR filters) with cut-off frequencies of:
#16, 50, 160 and 500 Hz (-6 dB per octave) were used to extract the envelope

###envelope signal was then used to modulate white noise, which was then 
#spectrally limited by the same bandpass filter used for the original signal

So how should the result sound? It should be something between hoarseness, a noisy cracking, but not so much robotic. It would be good if the dialogue would remain to some extend intelligible. I know - it's all a bit subjective, but don't worry about that - wild suggestions and loose interpretations are very welcome.

References:

Improve this question
$\endgroup$
9
  • $\begingroup$ One simple approach would be to modulate, hence multiply, the voice with the (noise+1.0). But another question: What are you trying to do? What is your overlying goal, when making voices unintelligible? $\endgroup$
    – Dominik Seibold
    Commented Jan 17, 2012 at 17:51
  • 1
    $\begingroup$ Why doesn't simply doing noisy <- audio + k*white_noise for a variety of values of k do what you want? Keeping in mind, of course, that "intelligible" is highly subjective. Oh, and you probably want a few dozen different white_noise samples to avoid any coincidental effects due to false correlation between audio and a single random-value noise file. $\endgroup$
    – Carl Witthoft
    Commented Jan 17, 2012 at 17:58
  • $\begingroup$ Ultimately I want to parametrically decrease the reliability of auditory information, so that the accuracy judgements will differ for different levels of manipulated audio clip. The accuracy judgement will be for the emotion - whether conversation is happy or angry. The problem is that it is extremely difficult to manipulate emotional content of a long speech utterance (like my clip attached above). People do it with a single vowels, but not the whole sentences. So I decided to generalize the question, and figure out the way to parametrically degrade the entire spectrum of audio information. $\endgroup$
    – Geek On Acid
    Commented Jan 17, 2012 at 18:21
  • $\begingroup$ @CarlWitthoft Your solution only adjust the amplitude of noise, and as I said - I need something that mixes noise with the signal. +1 you the suggestion that I need different samples of white noise - that might indeed make a difference as you pointed out. $\endgroup$
    – Geek On Acid
    Commented Jan 19, 2012 at 14:16
  • $\begingroup$ Well... I plead ignorance here: what is the mathematical definition of "mixing" two audio streams? I was naively assuming that, leaving out the existence of a programmable filter, all you can do with two vectors of time-sampled amplitudes is add them. $\endgroup$
    – Carl Witthoft
    Commented Jan 19, 2012 at 17:00

2 Answers 2

Reset to default
11
+50
$\begingroup$

I read your original question and wasn't quite sure what you were getting at but it's quite a lot clearer now. The problem you have is that the brain is extremely good at picking out speech and emotion even when the background noise is very high which is your existing attempts have only been of limited success.

I think the key to getting what you want is to understand the mechanisms that convey the emotional content as they are mostly separate from those that convey the intelligibility. I've got a some experience around this (in fact my degree dissertation was on a similar subject) so I'll try and offer some ideas.

Consider your two samples as examples of very emotional speech, then consider what would be an "emotionless" example. The best I can think of right now is the computer generated "Stephen Hawking" type voice. So, if I understand right what you want to do is understand the differences between them and figure out how to distort your samples to gradually become like a computer generated emotionless voice.

I'd say that the two main mechanisms to get what you want are via pitch and time distortion as a lot of the emotional content is contained in the intonation and rhythm of the speech. So, a suggestion of a couple of things that might be worth trying :

  1. A pitch distortion type effect which bends the pitch and reduces the intonation. This could be done the same way that Antares Autotune works where you gradually bend the pitch towards a constant value more and more until it's a complete monotone.

  2. A time-stretch effect which changes the length of some parts of the speech - perhaps the constant voiced phonemes which would break up the rhythm of the speech.

Now, if you decided to approach either of these methods then I'll be honest - they're not that straightforward to implement in DSP and it's not going to be just a few lines of code. You're going to need to do some work to understand the signal processing. If you know somebody with Pro-Tools/Logic/Cubase and a copy of Antares Autotune then it'd probably be worth trying to see if it'll have the effect you want before trying to code something similar yourself.

I hope that gives you some ideas and helps a little. If you need me to explain any of the things I've said any more then let me know.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your suggestions @Redeye. Time-stretch is unfortunately not an option, because there will be a condition in which I present them with the video information, so I need to keep the modified speech the same length as original one. Pitch distortion is an interesting approach - do you know any published references to explain this method better? $\endgroup$
    – Geek On Acid
    Commented Jan 24, 2012 at 16:48
  • 1
    $\begingroup$ Pitch shifting the speech to do what you want will involve two stages - firstly analysis of the speech to establish the current fundamental frequency profile, then secondly the pitch shift. The analysis is fairly straightforward and there's several methods which are effective. The pitch shifting is more complex - I'd try searching the AES journal for published references (JAES Volume 47 Issue 11 pp. 928-936; November 1999 looks like it might be useful). Like I said before, you're getting into some pretty complex processing here and it'd definitely be worth trying it with Autotune first. $\endgroup$
    – Redeye
    Commented Jan 24, 2012 at 19:12
  • 2
    $\begingroup$ Redeye has good suggestions, but I would just note that for pitch shifting speech, I would not recommend phase vocoder or any frequency domain approach -- PSOLA (pitch-synchronous overlap add) is a good way to go because it will sound better for a monophonic phase-locked instrument like the voice. $\endgroup$
    – schnarf
    Commented Jan 30, 2012 at 1:14
4
$\begingroup$

I suggest you get some music production software and play with that to get the effect that you want. Only then should you worry about programmatically solving this. (If your music software can be called from a command line, then you can call it from R or MATLAB).

One other possibility that hasn't been discussed is to completely strip out the emotion by using speech to text software to create a string, then text to speech software to turn that string into a robot voice. See https://stackoverflow.com/questions/491578/how-do-i-convert-speech-to-text and https://stackoverflow.com/questions/637616/open-source-text-to-speech-library.

To get this working reliably you will probably have to train the first piece of software to recognise the speaker.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I need filter the original files so text-to-speech is not really an option unfortunately, although I might think about some morphing paradigm between normal speech and synthetic speech. $\endgroup$
    – Geek On Acid
    Commented Jan 24, 2012 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.