Virtual Reality
Virtual Reality (i.e. put on a helmet and only see computer generated imagery) & Augmented Reality (i.e. overlaying computer generated images on real world images) are gaining a lot of traction. While VR & AR have existed for more than two decades, they have been both in a research stage and so expensive that they basically were non-existent when it comes to the average consumer. With the appearance of the Oculus Rift Kickstarter for VR and more and more powerful smart phones for AR, development in both areas is currently having a renaissance.
This document provides some limited background on VR and what to look out for.
Implementation
The Core SDK supports both Oculus Rift and HTC Vive out of the box but support for other devices can easily be added by you:
- Layouting: The layouting subsystem makes it easy to render the scene side-by-side for each eye
- Rendering: The OpenGL backend already supports lens distortion compensation
- Input: The input subsystem can be used to query the device and translate it to camera position/orientation information
- View Augmentation: By creating a so called augmentation, you can attach a separate tree of nodes to the scene graph that augmentations the original scene and environment. This makes it easy to render for example hand-tracking-visualization without having adapt the render backend itself.
Rendering for Virtual Reality
VR starts with a stereoscopic image so the system has to create two images, one for the left eye and one for the right eye. Both images are created from a slightly different position which produces the perception of depth. The distance between the view point of the two images is not arbitrary but instead has to match the interpupillary distance of the user. There is a little bit of math involved to make both projections align, but the important part is that the scene is rendered twice. And since one requires a high frame rate for VR, this means a) your GPU has to be very powerful or b) or scene has to be sufficiently simple.
One important element of VR glasses is that they all work with lenses. The lenses are what translates the screen to a decent field of view but they also introduce barrel distortion which has to be compensated by rendering the 3D content with a pincushion type of distortion, giving the characteristic look one often sees in screenshots. This is usually implemented as an image post-processing step in one of two ways:
- Render a fullscreen quad with a distortion shader
- Render a specifically distorted mesh with carefully chosen U/Vs to perform the distortion
Oculus for example used/promoted the first approach in its early development versions but now recommends the second solution. In a similar vain, lenses also create chromatic abbreviation, meaning that light is distorted differently based on its wave length. In plain English, the red channel is distorted slightly different than the green channel than the blue channel and again the rendering system has to compensate for that by actually rendering the colors misaligned.
Luckily with the advancement of the VR SDKs (e.g. OpenVR) a lot of internal details (render target distortion, timing, etc) have been abstracted and it has gotten much easier to achieve good results.
Overview of VR Devices
The following is an overview of the most popular consumer-level devices.
| Name | Category | Supported Systems | Tracking | Controller | Required Hardware | Comments | Support by Core SDK |
|---|---|---|---|---|---|---|---|
| HTC Vive | High-end consumer level | Windows, OSX, Linux | Room-scale tracking via two lighthouse boxes | Two hand controllers | High-end gaming PC | Builds on open standards | Native |
| Oculus Rift | High-end consumer level | Windows | Head tracking via camera mounted on display | Game Pad (optional hand controllers) | High-end gaming PC | Has its own ecosystem | Native |
| Sony VR | High-end consumer level | Playstation 4 | Head tracking via camera on TV | Two hand Playstation Move Controllers | Playstation 4, Camera, Move Controllers | None | |
| Samsung Gear VR | Mid-level 360 degree viewer | Android | Only rotation tracking | High-end Samsung phone | Developed in cooperation with Oculus | None | |
| Google Cardbox | Cheap | iOS, Android | Only rotation tracking | None | Mobile phone | None |
Note that even the current generation of high-end consumer devices have a screen quality that might disappoint novice users. Since the display is very close to the eye, the gaps between the individual pixels is visible which has been dubbed the “screen door effect”. Since the level of immersion is quit good, one usually forgets about it quickly.
The other important quality criteria is the level of tracking. With an HTC Vive, one can actually move in a room (provided one has enough space and doesn’t trip over the cable that goes from the headset to the PC). The Oculus Rift (and all other devices) on the other hand is optimized for a sitting experience. A camera – not unlike a webcam in form factor – is placed in front of the user and tracks upper body movement. Mobile-phone-based systems can only facilitate the sensors on the head of the user. The perception of realism is directly related to the level of tracking and so is also the comfort of using the device. For example, if the user moves through a room by actually standing and walking, your body will cope way better than if a game character moves but your body is aware that is sitting still.
Recommendations
What device to get depends heavily on your use case. For a quick, cheap wow-effect, get a google cardboard but be aware that since it only tracks rotation, the experience is limited. Oculus Rift currently seems to be the premier choice for gaming due to its own ecosystem and the number of exclusive titles. However it is under attack by Sony’s VR solution which has the benefit of building on top of the existing Playstation 4 which many gamers own already.
For serious VR however, we recommend the HTC Vive if you have the space for it. The room-scale tracking creates a completely different level of immersion and the impact of having hand controllers cannot be overestimated. The screen door effect seems to be less disturbing than on the Oculus Rift and due to the great immersion one forgets about it almost immediately.