0
$\begingroup$

I am trying to correct the orientation of a 360 degree camera frame using roll, pitch and yaw estimates coming from my C++ Extended Kalman Filter (EKF) code. I am interested in using libraries and packages such as OpenCV, Nvidia Visionworks or OpenVR. I am working on Ubuntu 16.04. Due to the requirements of the project, I am looking to get it done on GPU.

To give more context, I have a camera that provides 360 degree images which also has a set of IMUs integrated to it. There is an initial frame of reference and EKF calculates roll, pitch and yaw based on this frame of reference. As the camera changes the orientation, I want to use the EKF outputs to correct the orientation of 360 degree video frames coming from the camera.

1) So far I haven't seen any APIs or libraries providing this kind of a function for 360 degree cameras. I am looking for suggestions.

2) If this is not readily available, building a function from scratch using CUDA programming a good approach?

Edit based on the comment: The camera is mounted on a mobile platform which moves, thus changing the orientation (roll, pitch and yaw) of the camera. The initial orientation of the camera is used to fix a global coordinate system. The change in the orientation of the camera, as the platform moves, is calculated by estimating roll, pitch and yaw with reference to this global coordinate system. As the orientation of the camera changes, the orientation of the scene in a frame also changes. The idea is to align the frame back to global coordinate system so that all scenes are in alignment.

As an example, there is an idea to fix a global coordinate system where (x, y, z) will indicate the position of each pixel of the 360 degree video. It can be imagined that the pixels are lying on the surface of the sphere and the origin of the global coordinate system is at the center of this sphere. As the camera changes the orientation, this sphere rotates. Thus a pixel which corresponds to a particular object X will change its coordinates after rotation. Another way of looking at it, is that the local coordinate system has deviated from global coordinate system by roll, pitch and yaw. The objective is to create a rotation matrix from roll, pitch and yaw values estimated from EKF algorithm to rotate this sphere back to original position and bring it back in alignment with the global coordinate system.

I am looking for a function which is capable of doing this using GPU.

Improve this question
$\endgroup$
5
  • $\begingroup$ Can you talk a little bit more about "orientation" in the case of a 360 image? Can you give an example of how do you see this working? $\endgroup$
    – A_A
    Jun 28, 2018 at 5:48
  • $\begingroup$ @A_A I have added an edit. I hope this calrifies and answers your questions. $\endgroup$
    – skr
    Jun 28, 2018 at 13:15
  • $\begingroup$ OK, I was struggling to see where CUDA comes in, but I guess that your camera returns raw images and so lines->arcs so what you really want is to "scroll" the pixels on the arcs rather than scroll up / down by the reverse of the pitch/yaw. $\endgroup$
    – A_A
    Jun 28, 2018 at 15:11
  • $\begingroup$ I am not sure I understood that comment correctly. I do require the ability to correct using pitch/ yaw as well. I didn't quite understand scrolling on the arcs. Is it equivalent to changing yaw? $\endgroup$
    – skr
    Jun 28, 2018 at 15:14
  • $\begingroup$ No worries. That is fine, the confusion was mine. $\endgroup$
    – A_A
    Jun 28, 2018 at 15:24

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.