The purpose of an anti-alias filter is to remove signal and noise from frequency bands that would otherwise "fold" or alias into the band of interest during the sampling process. This can occur when going from an analog signal to digital (and thus would be the filter just prior to the A/D converter), or when resampling a digital signal such as done in multi-rate digital signal processing where a sampling rate is changed completely in the digital domain.
That said, the design of the anti-alias filter depends on the signal bandwidth of interest and how much of the Nyquist zone in occupies (see further description below on Nyquist zone). In order to realized a practical filter design (a brick-wall filter cannot be realized), the bandwidth of the signal of interest must always be sufficiently less than the Nyquist zone. A rejection requirement is established knowing the worst case interference (noise and other signals) that exist in the other Nyquist bands; note that all the spectrum that passes through to the A/D Converter in other Nyquist bands will alias multiple times on top of the digital spectrum. Ultimately the A/D itself will have analog input bandwidth acting as a low pass filter to keep the otherwise infinite folding from occurring. An initial misconception many make when first approaching this subject is to dictate the rejection based on not allowing any aliasing into the digital spectrum and thus requiring full rejection right at the Nyquist boundary. This often over-constrains the filter design, as signals just beyond the Nyquist boundary will indeed alias in, but they will not land within the signal bandwidth of interest and therefore can be further filtered out digitally later downstream where such filtering can be done much more effectively than in the analog domain. This is clearer in the plot below that shows that the Nyquist boundary is placed mid-way in the transition bandwidth of the filter; full rejection is applied at the point where an interfering signal if present would alias into the signal bandwidth of interest. The plot below shows the filter as it would appear in the digital spectrum: if this was a signal from the first Nyquist zone in the analog domain then this would be an analog low pass filter centered on DC (f=0). However this could also be a bandpass filter centered on any arbitrary frequency at a higher Nyquist zone, which also shows how good frequency planning (of signal of interest and sampling rate) is to place the signal, when real, in the center of a Nyquist zone, thus optimizing the filter realization by maximizing the allowable transition band.

This approach is also done in digital resampling applications; for example, a digital anti-alias filter must precede a decimator (throwing away samples to reduce the sampling rate). Multi-band filters are often used in this application, again applying rejection only at the frequencies that would fold into the signal bandwidth of interest and ignoring the rest to be further filtered downstream at the lower sampling rate where such filtering can be done more efficiently.
The bottom three diagrams help illustrate the aliasing mechanism and the idea of "Nyquist Zones". Once the aliasing mechanism is understood, then the design considerations for the anti-alias filter are clearer for signals of interest in any Nyquist zone.
All three plots show the frequency spectrum of the analog signal of interest on the top line, the digital sampling spectrum on the middle line (sampling is multiplying by a stream of impulses in the time domain, and the Fourier Transform of a stream of impulses in the time domain is a stream of impulses in the frequency domain, spaced by the sampling rate). The bottom line is the convolution of the two, resulting in the digital spectrum. The digital spectrum is unique from $-f_s/2$ to $+f_s/2$ where $f_s$ is the sampling rate, thus it is typically only shown over that domain (I find it helpful in many applications to extend it to $+/-\infty$ as I do below, showing the repetition over the $f_s$ interval).
The first plot show the case where the signal of interest is in the "first Nyquist zone" which is the most common form of sampling an analog signal-- the analog signal is entirely below $+/-f_s/2$ and therefore the frequency of the signal in the analog domain maps directly to the frequency of the signal in the digital domain.

Note that I will explain this all with complex signals: $cos(\omega_o t)$ consists of a positive and negative frequency consisting of $\frac{1}{2}(e^{j\omega_o t}+e^{-j\omega_o t})$. Once you see the cosine wave as two impulses in frequency each being a transform of the time domain exponential sinusoid ($e^{j\omega_o t} \leftrightarrow \delta(\omega-\omega_o)$, the rest becomes a lot more intuitive and covers all cases. Every single impulse in the frequency domain corresponds to a single spinning phasor in the time domain; so negative frequencies are phasors in time that rotate clockwise while the positive frequencies rotate counter-clockwise.
As a analog signal begins to cross the Nyquist boundary, it's content will roll around (alias) to the other boundary. For example if a signal is sampled at 10 Hz, the Nyquist boundary is +/-5 Hz. If the analog signal was $e^{j4\pi t}$ which is +4 Hz, it will be at +4 Hz in the digital domain (and at ...-16 Hz, -6 Hz, +14 Hz, +24 Hz,... etc). If this signal frequency was increased to +4.5 Hz it would digitally be at (...-15.5, -5.5, 4.5, 14.5....). Notice the signal approaching the -5 Hz boundary and what occurs when we increase the signal to +5.5 Hz: (...-14.5, -4.5, 5.5, 10.5....) Aliasing!
This is what causes a "spectral inversion" for signals sampled in the second Nyquist Zone as shown in the plot below:

As the frequency of the signal extends higher into an odd Nyquist Zone, the same digital spectrum is produced!

Thus we see how all higher frequencies will fold or alias into the digital spectrum. Therefore we must pre-filter the analog spectrum PRIOR to sampling to reject any signals (and noise) that will exist in these unwanted bands so that only our band of interest maps to the digital spectrum. (Note this could be the first Nyquist zone as in traditional sampling where we say the sampling rate must be greater than twice the highest frequency of interest, or undersampling applications where the anti-alias filter is a bandpass filter about a signal of interest in a higher Nyquist zone.