If a first-order IIR will do, modify that slightly, and you're done.
So the usual first-order low-pass filter can be defined as $y_n = h(\theta_n)$ such that $y_n = y_{n-1} + a(\theta_n - y_{n-1})$. This works great for $\theta_n \in \mathbb{R}$.
You want a low-pass filter that's defined on an interval that spans $360^\circ$. For reasons that will become apparent in a moment, you want $\theta_n \in \left [-180^\circ, 180^\circ \right)$, where addition wraps around* sorta-kinda modulo $360^\circ$, i.e., $170^\circ + 30^\circ = -160^\circ$.
For want of a better notation, let $\mathrm{wrap_{360}}(\phi)$ be a sorta-kinda modulo function** that maps the real number line onto the interval $\left[-180^\circ, 180^\circ \right)$ modulo $360^\circ$, where
$$\mathrm{wrap_{360}}(\phi) = \phi - 360^\circ \left \lfloor \frac{\phi + 180^\circ}{360^\circ} \right \rfloor$$
Then, the desired filter becomes $y_n = h(\theta_n)$ such that $$y_n = \mathrm{wrap_{360}}\left ( y_{n-1} + a\, \mathrm{wrap_{360}}(\theta_n - y_{n-1}) \right)$$.
The action of this filter is that for angles that stay close enough to $0^\circ$, the $\mathrm{wrap_{360}}$ function just acts as a passthrough. But for angles that get close to wrap-over (or for angles like yours, that start by being expressed in the range $\theta \in [0^\circ, 360^\circ)$) the actual angular difference is computed into something that expresses what it is physically -- that is, the difference between $1^\circ$ and $359^\circ$ is just $2^\circ$, not $358^\circ$.
Notes
- If your filter starts $180^\circ$ off, it will be at a false, but thankfully unstable, equilibrium. It may take an extra-long while to settle from startup.
- If your input angle is in some finite range around $0^\circ$, like $350^\circ$ to $10^\circ$, just recast it to $-10^\circ$ to $10^\circ$ and have fun.
- I can see a couple of ways that this notion could be extended to higher-order IIR filters, but it makes my brain hurt. If you need to do that, ask a separate question and someone will go to the effort to help you.
- If you want to use a FIR filter, you could first take your time sequence and "unwrap" it (Scilab and numpy both have "unwrap" functions that work on vectors, so I'm sure Matlab does to), then apply your FIR to the unwrapped version, then do whatever modulo operations you wanted to get it back to your desired interval.
- Alternately, you could do some sort of histogramming to determine the rough center of your average, shift everything, do the FIR, and shift back -- I think that would be computationally more expensive than unwrapping, though.
- Back when we actually had to worry about processor clock ticks, it was really convenient to map a 360 degree rotation into a 16- or 32-bit word, depending on the processor. On nearly all processors these days, in C, with a native $n$-bit word, if
a and b are int, c = a + b; implements $c = \mathrm{wrap_{2^n}}(a + b)$. I always experienced an underlying thread of glee when I wrote code that took advantage of this, because usually integer overflow is the bane of your existence when you're doing fixed-bit signal processing. Actually making it work for me made me feel like Tom Sawyer in the whitewashed fence story.
* I should be enough of a mathematician to tell you what such an entity is. I'm not. I don't think it counts as a field -- so we're stuck with lots of extra words.
** I just had a conversation with a mathematician who tells me that it's not a sorta-kinda modulo function: it's a plain old modulo function with a choice of equivalence classes that leads to the result being in $[-\pi, \pi)$ (I didn't want to offend him with degrees, so I talked in radians). Or more formally, that I'm just working with the additive quotient group, $\mathbb R/2\pi\mathbb Z $, with representatives chosen from $[-\pi, \pi)$, and I should just use the usual $\mod 2\pi$ notation. I'm not going to edit my text -- but there is terminology for this.
