For someone who has some experience in software development, loves his synths and knows the [very] basics of digital audio representation, what are the first steps he should follow in order to program a music synthesizer, DSP-wise?
Get Real Sound Synthesis for Interactive Applications, and download the Synthesis Toolkit. That will give you practical introductions and quality source code for most digital synthesis methods in use. Then pick whichever method you find most interesting, and start modifying the corresponding STK code to do what you want. Then try building something similar from scratch.
In addition, like hotpaw2 suggested, anything from JOS is also great introductory material. In particular, Introduction to Digital Filters with Audio Applications and Mathematics of the Discrete Fourier Transform both start from the basics, and don't assume a lot of background.
From a digital filter/effects perspective, another good starting point is to check out Faust. You don't even have to learn their block-diagram signal processing language, you can just use the examples to dump out VST, LADSPA, etc. template code as quick way of getting something running [which you can then modify].
That being said, it's even easier to start out with sound rendering, which is basically the same thing, but without worrying about real-time operation or interactivity. You just save wav files to disk. It's a great way to experiment with new techniques before going through the trouble of making them run quickly.
First you need to decide what kind of synthesiser you want to build - additive, FM, sample-based, etc. You also need to decide whether you want to emulate some kind of existing analogue synthesiser or just design your own.
The rest is fairly easy - you just need to implement the various synthesiser building blocks (e.g. oscillators, filters, noise generators, envelope shapers, etc) in software and then implement a way of "wiring" these together and controlling their parameters.
See Hal Chamberlin's book, Musical Applications of Microprocessors, which is a good introduction to many of the basics.
I would read an introductory book on DSP plus some books on the subject of computer music (Amazon and other bookstores list several). There's also a wealth of course material from Stanford on DSP and sound synthesis. Books on the human auditory system and psychoacoustics might also be helpful.
I'm quite surprised nobody mentioned SynthMaker yet. It's ridiculously easy to use, and though it makes it difficult to go low-level (be sure to use developer mode), you can learn a lot about Synthesizer architecture with it, and also about general DSP. And you can actually make use of the things built with it, in fact, I use it to construct almost all of my plugins.
As some of the other contributors have mentioned, a good primer on DSP is a must. This is a great book> http://www.dspguide.com/ and the author has kindly provided a full and free PDF download. Another common music tech course book which gives good high level views of various synthesis techniques is> http://www.amazon.com/Computer-Music-Tutorial-Curtis-Roads/dp/0262680823. I would also reiterate what the previous contributor suggested, Synthmaker, which is a very convenient visual development environment for developing VST plugins. Finally, if you just want to experiment without going to deeply into it you could try using Csound which is a programming language specifically designed for sound synthesis.
I just want to chime in since most of the answers focus on DSP issues. If you don't use a block design system like SynthMaker, you are going to spend a lot of development time in dealing with voice management. If you were writing a VSTi instrument plugin from scratch, the learning curve is much longer than for effect plugins. You will have to manage each note object, decide what happens when there are 20 notes on if your DSP is only fast enough for 16 voices, respond to pitch bend, portamento, modulation, etc. This is no easy task and that's why I highly recommend starting with Buzz or SynthMaker, and rolling your own when you finally come up against a wall.