Nyquist criterion for zero ISI states that in the frequency domain, the replicas of the frequency characteristic of combined:
TX pulse shaping filter - channel - RX pulse shaping filter
must add up to a constant value in order to ensure zero ISI.
How does it work in case of rectangular pulse signalling when the communication channel has an infinite bandwidth?
As I understand, the combined spectrum is then just a rectangular pulse spectrum (sinc function in the frequency domain) and I do not see how its replicas can add to obtain a constant value over the frequency. But in case of infinite channel BW, the pulses will not be smeared in the time domain, so ISI should not occur.