The role of the modulator in your block diagram is to make an analog signal, while in classical communications, the data and the channel coder is discrete. So no, in the sense of your block diagram, that can't work.
Note that you stated trouble because your signal wasn't binary in your approach: Not all channel coders work on binary symbols. For example, Reed-Solomon Codes are used exactly because they work on symbols from a larger field than the binary numbers. (That brings advantages when dealing with burst errors.)
Non-binary LDPC codes exist, and there's other codes, too, that don't use binary numbers. Generally, very little in our modern communications is "forced" to be binary, it just happens so that $\mathbb F^2$ is a very easy-to-deal-with finite field.
From the top of my head, I can't think of a single channel coder type which couldn't exist as a base-3 instead of base-2 algorithm, for example. (Building a good and performant channel decoder might be hard, though.)
TL;DR:
Your modulator needs to stay between your digital processing, which includes the channel coding, and the analog channel, but the notion that "digital == binary" is plain wrong.
Wth the very widely used Reed-Solomon codes you have an excellent counterexample to that, and you might want to look into these codes.
