My design has symbol rate and sample rate to DAC fixed but the ratio is not an integer. For example assume my symbol rate is 3 Mbps but the sample rate to DAC is 20 Mbps. So i am having 3 MHZ and 20Mhz clock. Do I have to use a polyphase resampler in the Tx path to bring 3 Mbps to 4 Mbps and interpolate 5 times to 20 Mbps by RRC and then send it to DAC? Similarly on the receive side? or is there any better method?
The symbol rate is independent of the sampling rate. You can sample the symbols at any desired rate that is greater than twice the bandwidth of the symbol, where higher rates would simplify the required analog filtering after the DAC (so higher sampling is better in this regard and the initial ratios the OP is using are reasonable).
The bandwidth of each symbol is driven by the pulse shaping that may be used. With typical pulse shaping such as Raised Cosine, two samples per symbol would be a reasonable minimum sampling rate to use but would also require tighter analog filtering.
To see this, consider each “symbol” as it would appear as an analog waveform and observe how you can sample that any any rate as long as you observe Nyquist’s criteria to sample greater than twice the highest bandwidth of interest in the signal. Going from digital to analog with a DAC is this same process in reverse, in that how many samples do you need to properly reconstruct the desired waveform.
Assuming this design starts with symbols sampled at one sample per symbol at a 3 MHz symbol rate, typically this would be interpolated (upsampled by inserting zeroes and then filtered) where the filter would be the pulse shaping filter defined by the waveform transmitter specifications (often a root-raised cosine filter but must be defined). This upsampling could be matched to the transmitter sampling DAC frequency using polyphase resampling or done in stages. If in stages resampling to only 4 MBps would not be sufficient for pulse shaping given the explanation above, but it would be reasonable to resample to 10 MHz for pulse sampling and followed by 2x interpolation for the final DAC sampling.
Not to dissuade you from using integer ratios when you can (and you certainly can here), but to let people know that there is nothing mandatory about using integer ratios, despite the approach being the staple of DSP textbooks throughout decades—there is no actual requirement for integer ratios. Consider that resampling is conceptually equivalent to playing audio out a DAC at one rate, and resampling it by an ADC running at another rate. The lowpass filter of the DAC essentially calculates the continuous between-sample points, to be resampled by the ADC. There is no reason you can’t calculate the new-rate sample points from a software lowpass filter directly.
One method for implementing the lowpass filter is a windowed sinc function. The only requirement is that it’s a lowpass filter, but we generally want to preserve phase linearity and have a flat passband. I’ll use windowed sinc as an example because it’s popular, easy to compute, and has desirable qualities, but my main focus is the non-integer ratio aspect.
The advantage of integer-ratio conversion is that output samples are evenly spaced relative to the input samples. This means we can pre-compute a windowed sinc table with just the values we need. For a factor of four conversion, the table would have four sets of values corresponding to each step forward starting from one sample, before the next.
That is, for a factor of four, output sample #1 would correspond to the position of the first input sample, #2 with one-quarter way between the first two input samples, #3 half-way, #4 three-quarters way, and the cycle repeats—#5 coincident with the second input sample, and so on. So, 0, 0.25, 0.5, 0.75, step to next sample and repeat 1, 1.25, 1.5, 1.75, then 2, 2.25…always the same four positions between samples for the sinc table to be aligned to (and thus a simple application of a polyphase filter).
If you were to do this for an arbitrary ratio conversion, you would not have this simple repeating pattern. Let’s say the conversion factor is 3.9. Output sample #1 would be coincident with the first input sample, as before, but the following output samples would be spaced about 0.2564 from there. This means that there would be no output sample coincident with the second input sample, and the output samples between the second and third input samples wouldn’t be at the same fractional offsets as between the first and second input. 0, 0.2564, 0.5128, 0.7969, 1.0256, 1.2821, 1.5385, 1.7949, 2.0513…there is no simple repeating alignment of the table.
That's the problem with the table approach—it works well work integer ratio conversions, but isn’t suitable for non-integer ratios. Why do we use a table at all? Because it’s faster to look up the needed values from a pre-computed table that it is to calculate them on the fly, and each output sample requires many windowed-since evaluations.
One solution is to calculate an oversampled windowed sinc table, and interpolate arbitrary values between table values as needed. The accuracy is determined by the degree of oversampling of the windowed sinc table, and the quality of the interpolation. Windowed sinc is a relatively smooth function, so linear interpolation can be acceptable given a table with sufficient oversampling and dependent on your requirements.
Another reason you might not want to stick with integer ratios is when the sample rate might need small adjustments, such as locking to a physical clock source. A similar approach would be to upsample at a high integer rate then use a simpler interpolation between resulting intermediate output samples (note that you don’t necessarily need to actually produce ever sample at the higher rate, just the ones you need for further interpolation).
I hope this helps, conceptually if nothing else!