There is a bit of distortion (see Tissot’s indicatrix), especially at the top and bottom of the image, but this is due to the problem of projecting a sphere onto a plane. On the Nexus 5 (and other GPE phones), the Gallery application includes a feature that allows you to either view the resulting Photo Spheres as spherical panoramas or to create “Little Planets”/”Tiny Planets”, which are actually Stereographic Projections of the spherical panorama. I found the effect really neat, so I wanted to see if I could recreate the projection in MATLAB.

As a teaser, here’s the output for my code:

Click through to get more information on the MATLAB implementation.

I found some Flash code that performs a similar function to what I wanted to achieve, so I have to send a big thanks to nicoptere for providing his code for PIXEL BENDER #4 projections. Thanks to the examples provided in his code, I was able to translate the projection conversion into MATLAB.

The actual calculation of stereographic projections is well documented on Wolfram Mathworld’s Stereographic Projection page, but basically we only need the last bit dealing with the inverse formulas for latitude $latex \phi$ and longitude $latex \lambda$. We can then use that to map the points onto our Equirectangular projection produced by the Photo Sphere function to points on a Stereographic projection. I am including the MATLAB code for the function at the end of the post, but here are a couple of examples of the conversion.

First, just to illustrate how the stereographic projection affects the Tissot’s indicatrix, here is an example based upon a equirectangular world projection from Wikipedia user Eric Gaba (CC BY-SA 3.0):

As you can tell, the output is a function of the latitude and longitude, with the “lowest” latitude in the middle of the image and the “highest” outside the bounds of the projection. This means that Antarctica is in the center while Arctic Circle is not visible.

If we apply the equirectangular to stereographic projection function to my image at the top of the post, but sample from the highest latitude for the center of the output image (by inverting the viewing distance in the code below), we can create another visually compelling projection:

For anyone who is interested, here is the function used to generate the equirectangular to stereographic projections (“Little Planets”) in MATLAB. Note that this function requires the inpaint_nans function:

There are a number of GoPro editions, but the newest one are the Hero3+ Silver and Black. The Silver Edition is very similar to the GoPro Hero3+ Black Edition (Amazon), but omits a few features, including some very high-resolution video recording modes and a feature that GoPro calls “Superview”. This post talks about a way that I attempted to emulate their Superview mode in MATLAB and put together an adaptive aspect ratio function that allows one to change the image’s aspect ratio while maintaining “safe regions” with minimal distortion.

This function allowed me to resize 4:3 images and video, like this one:

To a wider aspect ratio, for example 16:9:

Click through for info on how I implemented the code.

Let’s say we have an original, 4:3 image or frame from a camera (GoPro in this instance):

If we want to convert it to 4:3 format, we have a couple of options. The simplest is cropping out the top and bottom of the image:

The issue with cropping is that it cuts down on the vertical field of view. We could alternatively scale the image in the horizontal direction (or compress in the vertical, the effect is the same):

As you can see, scaling causes distortion in the image that can be distracting, especially when there are known points of reference (like people) in the frame.

An alternative to cropping or linear scaling, is the GoPro Hero3+ Black’s Superview mode, which is intended to non-linearly scale the image to retain as much vertical image information while attempting to minimize the perceived scaling distortion, especially in the center of the image frame.

Here is a description of the GoPro Superview feature, according to GoPro:

SuperView is a new feature introduced with HERO3+ Black Edition which allows you to capture an immersive wide angle perspective. What this mode does is it takes a 4:3 aspect ratio and dynamically stretches it to a 16:9 aspect ratio. This can be a great choice because it uses the height of the camera’s sensor that you get with 4:3 meaning that you will see more of the sky and ground assuming you are pointed at the horizon. … The way it works is that the camera automatically stretches out the sides of the video to fit into the 16:9 frame. The center of the frame is unchanged, only the edges are adjusted.

This sounds as though there is 1:1 mapping of pixels in the center of the frame with gradual distortion/interpolation towards the right and left edges. This would maintain the total field of view and information content in the image, while attempting to conform to a widescreen aspect ratio.

Abe Kislevitz has a nice discussion of 4:3 to 16:9 footage conversion with the GoPro and specifically discusses a plugin called Elastic Aspect. I chose to do a similar transformation in MATLAB to see if I could reproduce the Superview effect in “post production” (after an image or video has been captured).

To do this, I created a 1-Dimensional mapping of the input pixels to the output pixels along the y-axis and remapped the locations of pixels that were not linearly mapped to “Safe Zones’ using cubic spline interpolation.

Let’s say I want to stretch GoPro image of my wife and I from 4:3 to 16:9 widescreen aspect ratio. Here’s the original 4:3 image:

Using linear interpolation, this would be the expected mapping (and resulting image). Note the distortion in our faces:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [50 50]);
[/code]

With cubic spline mapping with a very small “Safe Zone” that is linearly mapped (5% of the image width), we would have this mapping and resulting image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [47.5 52.5]);
[/code]

Notice the difference in perceived quality already? Now, let’s protect the center 25% of the image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [37.5,62.5]);
[/code]

Looking good! Let’s say our subject is off to the side of the image, we can explicitly choose those regions to “protect” by ensuring a 1:1 mapping to avoid any distortion for our subject. For example, here’s a picture of my cousin’s husband, located slightly off-center:

I can explicitly select a “Safe Zone” around his body to avoid distorting him further. Here’s the remapping and resulting image:


Created using the following code (from the below function):

[code lang=matlab]
outImage = bSplineResizeWidth([1080,1920], inImage, [], 1);
[/code]

So, although there is still inevitable distortion around the outside of the image, it seems as this may be an effective way to emulate GoPro’s Superview mode and perform an adaptive aspect ratio adjustment.

For the sake of discussion, there is another method called seam carving (commonly known as content-aware scaling with Adobe Photoshop), but this can introduce some artifacts, depending on the scene. It is also quite slow. Here is an example implementation in MATLAB by Danny Luong: Seam Carving for content aware image resizing: GUI implementation demo. In general, it appears as though seam carving is better suited for images rather that videos (note that the original authors of the algorithm developed a similar method for processing video):

Here is the code for both procedures:

http://www.youtube.com/watch?v=bLxYTT_0GEI

I tried my hand at processing the HDR video output and was able to get a reasonably nice tone-mapped video:

After the break, you’ll find how I processed the initial Magic Lantern video using MATLAB and exiftool and tone-mapped the output using Luminance HDR.

First, we need to process the video with a function I (poorly) named ‘Step1MovieToInterpolatedFrames.m’ to separate the dark and light frames. The video is first loaded using the VideoReader object. Then we check to see if the first frame is darker or lighter than the second. This is admittedly a bit of a hack, but given the gross differences in exposure, seems to work well enough. After determining whether the first is light or dark, we then loop through all the frames of the movie, saving the real frames and appending an “L” to signify they are “light” and, also, interpolate between the frames. Why go through the bother of interpolation? Well, there will be image registration problems with the tone-mapping if we assume that a given dark frame matches the earlier or later light frame, especially with high-speed motion. Interpolation helps us “smooth” these errors out. Note that ideally we would use a morphing algorithm (similar to the one used by Twixtor), but this is the quickest method for the time being. After saving each frame, I use exiftool to assign an aperture value. Note that this has nothing to do with the real aperture value, but helps Luminance HDR tonemap the composite image. We do this for the dark frames as well, but now we take into account the EV shift in the video’s ISO when writing the aperture value using exiftool.

The second function, ‘Step2FramesToHDRFrames.m’, takes the individual light and dark frames and generates tone-mapped images. We go through every frame and use the Luminance HDR CLI (command line interface) to generate an HDR image and tone-map it (here using the mantiuk08 tone-mapping operator).

And the final function (‘Step3HDRFramesToVideos.m’) compiles all of the tone-mapped images into videos (one for the light frames, one for the dark frames, and one for the tonemapped frames).

The code can be found at the bottom of the post.

So, what do each of the Luminance HDR tonemapping operators look like (with their default parameters) when applied to a video? Here’s the source (note that YouTube strips out the alternating frames, you can find the original MOV here):

Ashikhmin

Drago

Durand

Fattal

Mantiuk 06

Mantiuk 08

Pattanaik

Reinhard 02

Reinhard 05

I created a series of HDR cube maps using NVIDIA’s CubeMapGen (currently hosted on Google Code). Starting with the Debevec light probes, I applied a Gaussian blur with increasing kernel size (10°, 20°, 30°, 40°, and 50°), creating 6 cube maps (one for each blur). In the videos, the cube maps have increasing blur from left-to-right, top-to-bottom. Note that I did not tone-map or account for changes in overall exposure (so the specular reflections can appear blown-out, especially for the higher blurs). After the break, you can see the effect using different light probes (and different shapes).

Using the OpenGL/Psychtoolbox framework I have previously described, I replicated this interesting effect. When you play the following movie, a 3D sphere (with sinusoidal perturbations) is rotated. Note the axis of perceived rotation when the object has specular reflections (1st half of the movie) and when the environment map is “painted” onto the surface (2nd half of the movie).

The physical motion of the object is the same in both cases — the object rotates around the vertical axis. When the object only has specular reflections, it appears to rotate around an oblique 45° axis, but when textured it rotates around the vertical axis. After the break, I show similar effects when the object is rotated around the horizontal axis, 45° axis, and when the spatial frequency of the perturbation is manipulated.

Here is the same object, now rotated around the horizontal axis. We have the same perceived effect, rotation around the oblique 45° axis when only specular reflections are present.

Here is the same sphere, now rotated around an axis 45° off-vertical. Note that the motion for the specular case appears the same when the object is rotated around the vertical and horizontal axis (as shown above) or around the 45° axis (shown below). However, the motion is clearly different when the object is textured.

Neat, right?

Now, here’s a series of videos illustrating the effect for perturbations with different spatial frequencies. Note the changes in perceived object geometry and axis of rotation for low and high frequency perturbations.

2 8 more videos (with different shapes and illumination conditions) after the break.

In order to generate “organic” stimuli with smooth undulations, I needed to systematically manipulate the surface meshes of 3D spheres to create smooth peaks and valleys.

To generate 3D “potatoes,” I start with an icosahedron whose mesh is refined to approximate a sphere. The Bioelectromagnetism Matlab Toolbox has a function that is incredibly useful for producing icospheres in MATLAB:

• sphere_tri.m

sphere_tri.m requires the following subfunctions (also available in the toolbox):

• mesh_refine_tri4
• sphere_project.m

Calling sphere_tri with no recursive refinement, we get an icosahedron:

With further refinement (increasing nRecurse), we can approximate a sphere. Here’s nRecurse = 1:

nRecurse = 2:

nRecurse = 3:

nRecurse = 4:

nRecurse = 5:

I like to use nRecurse = 5 because it produces an icosphere with 10242 vertices (and 20480 faces), which is usually adequate to generate nice-looking “potatoes”. Increasing nRecurse to 6 produces a icosphere with 40962 vertices (and 81920 faces), which is usually more than necessary.

Once we have our icosphere, we want to apply some sinusoidal perturbations to change its shape. The easiest way is to apply a sinusoidal grating to each axis. First let’s define y as vertical, x as horizontal, and z as depth. First, here are some example gratings applied to the X-axis (note that I hold the amplitude of the perturbations constant for these illustrations, but change the angle and frequency):

Y-axis:

And Z-axis:

Note that the only thing changing from example to example is the line where the perturbation is applied to the shape’s vertices:

By stacking multiple sinusoidal perturbations, you can produce some nice “blobby potato” stimuli (please excuse the quality of these examples, they were generated using the MATLAB patch command):

After the break, see how increasing the size of the convolution kernel affects the quality of the 3D noise.

Note that “Time to Process” was calculated tic and toc (see above) on a quad-core Xeon W3530 @ 2.80GHz with 12 GB of RAM. In addition, the 2D FFT animations were generated using each frame of the GIF animations using the following code:

k		Time to Process	2D FFT
1		0.4167
3		0.8670
5		2.2663
7		5.7784
9		11.2293
11		20.8108
13		33.3277
15		52.0085
17		82.3220
19		187.6235
21		397.4730
23		615.1934
25		852.5891
27		1190.7
29		1641.1
31		1822.3

Subjectively, there is diminishing return as the convolution kernel increases past 13-15 pixels. Objectively, it makes little sense to spend the computational time processing past 15:

Here is a MAT file with these 3D noise arrays:
noise3Dwrap64_k1_31.mat