How to use rectangular window design to design a length-9 polyphase filter which delays an input signal by 3/5 of a sample? An expression for the 9 coefficients is required.
Just sample a Sinc function at the appropriate phase offset, and truncate the FIR as needed to meet your rectangular window length requirement. You only need 1 phase for a constant delay while keeping the same sample rate.
Since the question was how to specifically implement this with a polyphase filter, I offer the following:
Such a true polyphase filter structure could be done by designing the base FIR filter with 9*5 = 45 taps and then mapping this to polyphase using row to column mapping of the taps in the one 45 tap FIR filter to 5 9 tap polyphase filters. In this approach each filter out would be an additional 1/5 of the delay, so choose the 3rd filter to get 3/5.
See the Interpolator implementation at this link for further details on how to construct the Polyphase Filter: How to implement Polyphase filter?
Each output of the polyphase filters in the interpolator is a delayed version of the same signal (hence how interpolation can be performed with these structures). Each filter is an allpass filter with a different delay (hence "poly-phase").
A windowed sinc filter with 9 taps has an inherent delay of around 4 taps, so depending on the context this could be useless. A fractional delay using an allpass filter might be a better choice.