[ EXPERIMENT ON DUAL-FISHEYE VIDEO PLAYER ] posted March 07, 2010

One of the ideal approaches to shoot a full-sphere panoramic motion picture covering both hemispheres or all possible directions might be to utilize such all-out systems like the Ladybug™ by Point Grey Research or the DODECA 2360 Camera System™ by Immersive Media Corporation while they are considerably expensive for the ordinary panorama fans and enthusiasts.

As part of our efforts to deliver on affordable systems to shoot and publish panoramic motion pictures using non-professional products or resources so that as many possible individuals or small groups of people can enjoy the world of moving panoramic images, we have spent some amount of time to examine the feasibility of the "on-the-fly dual-fisheye stitching", an approach to dynamically stitch a pair of full-fisheye moving images covering all possible viewing directions, as shown in the following example:


The concept or approach of "dual fisheye" is already nicely summarized by iPIX™, Internet Pictures Corporation. Please visit http://www.ipix.com/downloads/Immersive_Imaging_White_Paper.pdf to download and view the online paper titled "iPIX Immersive Imaging" published by the company in 2002 for your reference.

What we have done this time is to develop a Flash™ based video player named PanoDualFisheyeFlashVideo_v30.swf which reads a video clip containing a pair of fisheye images as shown above, and dynamically stitches them into a normal (perspective or rectilinear) view as well as orthogonal and stereographic views. In this experiment, the dual-fisheye video clip was actually taken through the "Spherecam" system already reported in our exhibition page posted on December 14, 2009, while such dual-fisheye video clips can be obtained through several different approaches. The simplest way might be to shoot two separate video footages at different times by using a set of video cam system mounted with a full circular fisheye lens, and edit those two videos into a single video clip containing the pair of fisheye images, by using a piece of software tool such as "StereoMovie Maker" developed by Mucchan, T.Aizawa and Tomo et al., for example for Windows users. While the tool is actually designed to create several types of 3D movies in a very convenient way, we can leverage a capability of the tool for the purpose of concatenating two fisheye images side-by-side wise to be used as an input to our PanoDualFisheyeFlashVideo_v30.swf. The output of the StereoMovie Maker, which is .avi, needs to be converted beforehand into a format such as .flv or .f4v readable by the Flash based players.

A sample outcome from this approach can be viewed in the following video. In this example, PanoDualFisheyeFlashVideo_v30.swf actually reads the video in .flv format which contains a pair of full-fisheye images as shown above, and dynamically stitches those two fisheye images into rectilinear and other views covering all possible directions, delivering on a complete omni-directional motion pictures without using any dedicate and expensive shooting, stitching nor publishing system, software or hardware at all, as a result.
In addition, PanoDualFisheyeFlashVideo runs online as well as offline on any operating system including Windows, Mac and Linux.

That says PanoDualFisheyeFlashVideo is the most inexpensive, flexible, interoperable, manageable enabler allowing you to easily and practically shoot and publish full 360-degree motion pictures.



The dual-fisheye video player, PanoDualFisheyeFlashVideo_v30.swf, is already made available for free at our download page so that anyone can replicate the experiment and hopefully obtain better, improved or enhanced results.

Now, let's take a look at the XML code relevant to the stitching used in this example to see how the on-the-fly stitching works:
<play_objects>
. . . . .
<crop width_A="0.965" height_A="0.965" cx_A="" cy_A="" width_B="0.965" height_B="0.965" cx_B="" cy_B="" overlap_ratio="0.03"/>
<projection roll_A="" roll_B="" pitch_A="" pitch_B="" yaw_A="" yaw_B="" />
. . . . .
</play_objects>

In order to let the pair of fisheye images be stitched as seamless as possible, we have introduces several new parameters or attributes dedicatedly used by PanoDualFisheyeFlashVideo. The values for those attributes should be appropriately set to obtain an optimized result, while those settings need to be manually proceeded unfortunately, because the internal stitching mechanism implemented in PanoDualFisheyeFlashVideo is not that intelligent nor automatic but rather simple and straightforward.

As already shown above, the attributes newly introduced are: width_A, height_A, cx_A, cy_A, width_B, height_B, cx_B, cy_B, and overlap_ratio for <crop /> element; and roll_A, roll_B, pitch_A, pitch_B, yaw_A and yaw_B for <projection /> element.

The two sets of four attributes in <crop /> element: width_A, height_A, cx_A, cy_A; and width_B, height_B, cx_B, cy_B are used to specify two areas to be cropped from the two fisheye images, "A" for left and "B" for right fisheye respectively.

The default values for those attributes are:
width_A="1.0", height_A="1.0", cx_A="0.5", cy_A=”0.5";
and
width_B="1.0", height_B="1.0", cx_B="0.5", cy_B="0.5".

This means that you do not need to explicitly specify those values in a case where dual-fisheye images are neatly allocated in such way like two circular edges of the fisheye images are exactly inscribed by the rectangular frame of the video. You can eliminate those attributes from <crop /> element in this case.

Among all of the attributes newly implemented this time, the most notable one is overlap_ratio, which is used to define a relative value of the width of the seam to be overlapped across two fisheye images. This approach called "overlapped stitching" allows you to put together two fisheye images more smoothly and seamlessly than the older approach of single-lined stitching, while the overlapped approach degrades the playing performance as it needs more CPU power to carry out pixel-wise overlapping operation. Please note that the playing speed becomes considerably slower when you set the value of overlap_ratio larger than zero. Typically the value of overlap_ratio should be between 0.00 and 0.06.

The following figure illustrates how those attributes represent relative dimensions and layouts of the dual-fisheye images within a video frame.



Other six attributes such as roll_A, roll_B, pitch_A, pitch_B, yaw_A and yaw_B for <projection /> element are designed to be used to justify or tune up three dimensional direction of the cropped fisheye images that are used as textures mapped onto a sphere model virtually created in the player, PanoDualFisheyeFlashVideo. The values should be specified in degree. Even in the worst case, they should be less than 10 from the practical standpoint.
Now, let's take the following video frame as an example to describe how the mapping can be tuned up by using those attributes.



The default values of roll_A, roll_B, pitch_A, pitch_B, yaw_A and yaw_B for <projection /> element are:
roll_A="0", roll_B="0", pitch_A="0", pitch_B="0", yaw_A="0" and yaw_B="0".

Therefore, in an ideal case where the intended hemisphere images are exactly allocated within the two fisheye circles, meaning that if your shooting system including optical apparatus is perfectly configured in terms of capturing the image of three dimensional space, you do not need to explicitly specify the value of those attributes to perform any extra tuning up. In such case, you can eliminate these attributes from <projection /> element.

The following figure illustrates how those attribute affect the texture mapping behaviors when the on-the-fly stitching operations are carried out by PanoDualFisheyeFlashVideo. The figure actually shows a range of orthographic views of the sample video frame, with each of the seven views representing the case where one of the attributes is set to "20" and all others are set to "0". The value of "20" is not a realistic value but was used to blow up or exaggerate the role of those attributes. A range of our experiments tells that they should be less than 10, even in the worst case without intention.



We do hope you make the most of PanoDualFisheyeFlashVideo by leveraging those newly implemented options.
Thank you.
 
Operated by Ryubin Office, Kanagawa, Japan (http://www.ryubin.com/). Since February 9, 2007.