Problem Summary
You have a couple of difficulties with this problem:
- The microphone array isn't a uniform linear array (ULA), which is the most straightforward to model steering vectors for.
- Unless you have a scenario where the types of sounds you'll encounter have a small fractional bandwidth, you can't use standard phased array beamforming or direction finding (DOA) without first bandpass filtering to a narrower bandwidth (I assume the band in question is 20 Hz to 20 kHz, the human audible range). This is because phase delay beamforming rests on the assumption that the signal of interest can be approximated as having one frequency/wavelength.
Solution
This question deals with the exact same issue. To solve this problem, the Pyroomacoustics library uses the following steps, as I detail in my answer there:
- Filter the sounds into narrow frequency bins that have acceptably small fractional bandwidth.
- Perform MUSIC separately on each frequency bin, using phase steering vectors based on the array geometry, which can be arbitrary.
I explain in my answer how to calculate steering vectors for an arbitrary array:
Let $X_{array}$ be a 3-by-M matrix of 3-D coordinate locations for the M microphones, and $\hat D_{dir}$ be a 3-by-N matrix of N unit vectors that point in the directions of interest (e.g., perhaps equally spaced in azimuth relative to the array from -90 to +90 degrees). Then, the N steering vectors are given by: $$
V = e^{j\frac{2\pi}{\lambda}\cdot X_{array}^T \hat D_{dir}}
$$
where $V$ is M-by-N and $\lambda$ is the wavelength.
See the rest of that answer for more details. Note that in your case, the wavelength $\lambda$ will be different for each frequency bin.