Stereoscopic 3D Overview

Stereoscopic 3D API Overview

Stereoscopic 3D or "stereoscopy" is implemented on the EVO 3D using a parallax barrier in front of a liquid crystal display. This technology creates the glasses free 3D imagery by blocking each of the stereo pair of images from the opposite eye's field of vision, effectively halving the screen resolution to accommodate both the left and right images within the same display.

There are several image arrangements for images and video that define the left and right pairs of the stereo images. Both images can be arranged by left and right or top and bottom as a single image or frame each compressed horizontally to half the size so as to not to occupy a larger image size or the images can be interleaved.

The stereoscopy display is managed by hardware and can be turned on and off both programmatically or by applications that can detect if an image or video has 3D content, such as the default Gallery application. When recording there is a physical switch to indicate recording in 2D or 3D.

Optimal viewing distance and angles:

The EVO 3D specifications:
  • Processor: MSM8660 (1.2 GHz Dual Core processor) + SQN1210 (WiMAX)
  • Operating System: AndroidTM 2.3 with HTC SenseTM Internal Memory: 4GB eMMC / RAM 1GB
  • Display: 4.3" inch Super LCD 540x960 QHD resolution 3D display
  • 3D Capture and playback for photos and videos, display downloaded content formatted in 3D, stream 3D content wirelessly to 3D TV via DLNA or HDMI via microusb MHL port
  • supported 3D file formats: .jps, .mpo, .mp4, .3gp

How the Gallery application detects 3D content:

With still images, jps and .mpo are the extensions used to indicate a stereoscopic image. A jps file is simple a jpeg file with the L and R images side by side (SBS) in one jpg file. There is no header info to indicate the format and the format is just based on the extension. A mpo (multi part object) image format holds one or more jpg images sequentially in the file format, for stereoscopic L and R images they are the first 2 sequential images in the container.

The mpo format contains the same header as a jpg, therefore if an mpo file is simple renamed to use the jpg extension, the result will be a standard jpg image that displays only the first image with standard jpg image viewers. This allows to easily share the same file as a 2D or 3D image. Also unlike the jps format where L and R images are compressed horizontally to obtain same size ratio, the mpo format store each of the L an R stereoscopic images in the full horizontal resolution.

Preparing 3D Video with S3D metadata:

For preparing 3D video content, the standard H.264/MPEG-4 AVC Supplemental Enhancement Information (SEI) is used. This is metadata included in the video file itself. Specifically the frame packing arrangement (FPA) field. Using the side by side (also referred to as SBS) LR format would correspond to the frame packing value of 3. This is the recommended format for left-right video frame arrangement as not only is it the default format used when recording a video on a EVO 3D device, but it is also currently a common format found on many of the new 3D capable devices.

SEI FPA (Frame Packing Arrangement) message that contains the 3D arrangement:

  • 0: checkerboard - pixels are alternatively from L and R
  • 1: column alternation - L and R are interlaced by column
  • 2: row alternation - L and R are interlaced by row
  • 3: side by side - L is on the left, R on the right
  • 4: top bottom - L is on top, R on bottom 5: frame alternation - one view per frame

so for SBS LR arrangement use:

  • 3: side by side - L is on the left, R on the right

If you are encoding a 3D video, and your video encoding software does not support setting the SEI FPA field, you can re-encode the video using software that can set that appropriate field. One such example is the open source encoder called x264 which is part of VideoLan organization. Several commercial encoding software solutions are also adding the ability to set this field.

example command line usage: x264 --frame-packing 3 -o output.mp4 input.mp4 (note: this encoder has many additional parameters so default options are applied)

Useful resources:

Programmatically enabling and disabling S3D display setting in applications:

As a part of the HTC Open Sense SDK, The DisplaySettings class is provided to allow applications to enable or disable S3D on a particular surface.

It can support several stereoscopic formats:

  • STEREOSCOPIC_3D_FORMAT_OFF
  • STEREOSCOPIC_3D_FORMAT_SIDE_BY_SIDE
  • STEREOSCOPIC_3D_FORMAT_TOP_BOTTOM
  • STEREOSCOPIC_3D_FORMAT_INTERLEAVED

The following static method is used to set the S3D mode:

DisplaySetting.setStereoscopic3DFormat(Surface surface, int format);

Where format is one of the formats listed above (for SBS, LR ia assumed) and surface is the current SurfaceView's surface, as used when drawing on a Canvas or a GLSurfaceView when programming with OpenGLES.

It is important to note that in either case, what you will be providing to the displayable surface that has S3D enabled is either existing images or video frames recorded in 3D or two LR programmatically generated images taking into account the mathematics involved with providing two camera angles with a certain distance between them.

So when painting on a canvas for example, you will need to paint twice, one for each eye's point of view. With OpenGL games, two view ports will be required with the world drawn twice, once for each view port.

In the provided sample code, a stereoscopic 3D image is provided as a resource. This viewed stereoscopic 3D image which measures 960x540 and could have been created by two LR side by side images, where each image's width is half of its respective original 960x540 size. So when viewed without the S3D display setting enabled, each LR image actually measures 480x540.

Additionally, sample code for S3D OpenGL consisting of a touchable spinning cube (with pinch to zoom) is also provided.

Note: currently DisplaySetting.setStereoscopic3DFormat(Surface surface, int format); does not affect video and camera surfaces as they are overridden by the video content and camera selection. The video surface is driven by the presence of the SEI FPA flag described above. The camera selection affects S3D display as follows:

Camera selection:

  • Camera 0 = Camera.CameraInfo.CAMERA_FACING_BACK (as of Android 2.3)
  • Camera 1 = Camera.CameraInfo.CAMERA_FACING_FRONT (as of Android 2.3)
  • Camera 2 = undefined in Android platform, HTC specific value for stereoscopic camera mode (both rear facing cameras)

so the equivalent to the hardware 3D-2D switch for the camera mode would be to switch between camera selections 0 (also the default) and 2:

Camera.getCamera() and Camera.getCamera(CAMERA_STEREOSCOPIC) where CAMERA_STEREOSCOPIC = 2

Note: for more info on the four S3D examples see the Sample Code section.

3D content recommendations:

For best results with 960x540 or larger with same ratio (e.g. 720dp)

A few 3D content best practices:

  • keep object in the frame (avoid the edges)
  • avoid fast motion across screen
  • avoid eye fatigue (refrain from popping out of screen)

S3D Formats:

Video Formats:

  • EVO 3D records into a .mp4 file container. Example stream info:
  • Video Track: 1280x720, ~30.013 fps, 5.18 Mbps (LR side by side in each frame)
  • Audio Track: MPEG-4 Audio stereo, 44.1 kHz, 64 kbps

Image format:

1920x1080 MPO container (L and R sequential jpeg images) Renaming the ,mpo to .jpg will provide L image readable by any jpeg image viewer.

The way that the EVO 3D (and YouTube) can detect that it is a mp4 with 3D content is by using the standard Supplemental Enhancement Information (SEI) metadata information embedded in the mp4 file.