# Image Processing: 360x180 panorama from discrete images

I'm coming from math as I was told that this would be a more fitting place for this question.

For the last few weeks, I have been trying to create a full 360°x180° spherical panorama from distinct images taken with the same camera (as Google Street View does it). I'm trying to use as few libs as possible because I want to understand all the math behind every step of the algorithm.

Apart from a lot of research papers (this one for instance https://courses.cs.washington.edu/courses/cse576/05sp/papers/MSR-TR-2004-92.pdf, I have found many of them but they all seem to quote or use part of this one), I've also been disecting Google Street View, PTGui, Photoshop's Photomerge, Hugin, etc. in my quest.

This is going to be a bit long even if I tried to make it concise... So thank you so much for bearing with me here :)

Given:

• Pictures taken from the same phone: known focal length, fov, resolution, etc.
• All pictures, taken together, cover enough surface to create a full 360x180 panorama. Tilt angles at the moment of capture are unknown.
• Known pairing of pictures

Here's the short version of what I am doing:

• I implemented Harris feature detector to get feature points coupled with brute force correlation matching to get matches for each pair of picture.
• Then feed that result into a RANSAC algorithm to find the best homography for each pair of picture.
• Choosing a central picture, I multiply homographies using the known pairing of pictures to get the final transformation matrix for each image
• My rectilinear panoramas look great with small field of views; with greater field of views (> 100°), the stretch is unusable - which is totally normal without a cylindrical / spherical projection.

• This is where equirectangular projection comes and I'm having issues with it. I think I might do something wrong here and misunderstand something, hope someone can help me. Here's my understanding of it:

• For each picture $P_i$:
• I project 8 keypoints $[xKey_{i,j}, yKey_{i,j}]$ (4 corners & 4 centers of each edge) using the homography corresponding to this picture, becoming $[xKeyProjected_{i,j}, yKeyProjected_{i,j}]$
• Knowing focal length f and setting s=f, I apply a spherical warp $$xKeyWarped_{i,j} = s*Atan(\frac{xKeyProjected}{f})$$ and $$yKeyWarped_{i,j} = s*Atan(\frac{yKeyProjected}{\sqrt{xKeyProjected^2 + f^2}})$$
• Getting the min and max values for j = {1...8} give me a bounding rectangle $Rect_i = [xMin_i, yMin_i, xMax_i, yMax_i]$
• Taking the min & max values of $xMin_i$ $yMin_i$ $xMax_i$ $yMax_i$ for all picture index $i$ give me a projection surface $[mergeWidth, MergeHeight]$ (and an offset to center)
• I will now proceed to the merge:
• For each pixel on the blending surface $[xBlend, yBlend]$, check in which bounding rectangles $Rect_i$ it fits
• (If it belongs to multiple pictures, for now I select the first picture that fits, I will apply a Seamless Masking later on)
• Do the reverse process: Inverse Spherical Projection & Reverse Homography corresponding to the picture that fits to get the pixel of the original image $P_i$

Results are shown in this album - I would have loved to link to individual images but it's my first time participating so I'm not allowed to :(

http://imgur.com/a/7FTCl

• ~100°x~100° panorama: Pretty cool, it seems to work very well where rectilinear did not even give me a result because it crashed, trying to create a 16000x16000 picture.
• ~60°x180° panorama: Again, a great result
• ~180°x60° panorama: But here, not so much... the pictures that are heavily distorted by homographies in a flat panorama now become somehow upside down.

Which lead me to thinking a bit more about what I read in papers: homographies are not the way to go for > 100° panoramas. So I just wrote a second version of my Ransac algorithm that returns Affine transformation instead of Homographies.

Results (refer to the same album):

• ~60°x180° panorama: Where's my equirectangular warping ? :( Also, the errors are adding up quickly
• ~100°x100° panorama: It looks pretty bad where Homographies gave much better results
• ~180°x60° panorama: But in this scenario, it gives great results!

So... what am I doing wrong ? Is there something wrong in my way of thinking or is it an implementation bug ? Should I use Affine or Homography ? I think I'm overheating from math overload, so thank you in advance for ANY help!