Eye tracking Virtual Reality

In the Blink of an Eye – Eye Tracking in VR6 min read

Latest posts by Dmytro Ternovyi (see all)

For many people, the idea of Virtual Reality has been nothing but science fiction for years, relegated to badly-rendered effects in movies and television. Now that VR is a commercially-available technological reality, inventors and consumers both are starting to investigate and push the boundaries of what the technology can accomplish, in gaming and beyond, especially with new features that can improve gaming experience. One area showing the most interest is that of eye tracking in VR hardware.

Getting a Handle on VR

While virtual reality allows users to experience a convincing visual and auditory representation of environments both realistic and fantastical, in order to interact with those environments, users still need to use  hand-held wands or other control apparatus. VR users can “look” around their virtual environment by moving their head in real life, but the tracking is limited to field of view, and any commands in games or other apps have to be made manually.

Voice commands are a possibility (who doesn’t want to feel like a Star Trek captain on the bridge of their ship), but so far they aren’t well implemented, facing issues of usability (and user error). Still, VR gear manufacturers continue to search for alternatives that will improve the user experience and provide even more functionality to games and other VR applications.

The Lowdown on Eye Tracking

One way to do this is through the use of eye tracking. Eye tracking is exactly what it sounds like; a process where the line of sight of the user is tracked using cameras and software. The type of eye tracking used in virtual reality, optical tracking, works by reflecting infrared light off the surface of the viewer’s eye, and tracking changes in that reflection over time, using landmarks inside the eye such as the pupil or cornea.

Really Reality?

Eye tracking in VR uses this information to extrapolate what the viewer is looking at inside the virtual space being projected, and then uses that information to perform actions as programmed. Eye tracking in virtual reality has a number of potential uses. In video games, immersion is everything. The ability for a virtual reality game to make the user feel like they are actually in the game they are playing can make the experience more intense, more delightful, or even more terrifying, depending on the game being played.

Imagine you’ve found yourself in a creepy haunted house, your only source of light a hand-held flashlight that keeps flickering out. Now imagine you come across a mannequin in a corner. You could swear it keeps moving every time you’re not looking directly at it. Eye tracking can make those kinds of encounters a reality in ways that have previously been impossible.

When walking through a busy, medieval-era market, you could actually make eye contact with passing NPCs, or non-player characters, potentially setting off new types of encounters or revealing content. Games designed around what you’re looking at could become a booming industry; what if instead of just looking for Waldo in a 2-D rendered image, you could actually search for him in a 3D rendered environment, with people and objects reacting to what you’re actually looking at?

Foveal – Foveated Rendering

More than merely improving the functionality of a game, however, eye tracking also has the potential to allow for more complex rendering on current hardware. Eye tracking allows for a type of visual projection called foveated rendering, a process which uses the eye tracking to mark the portion of your field of vision which is completely in focus at any given time.

In real life humans have a small round focal point at the center of our gaze, and outside that region, most things are fuzzy and out of focus. This carries over into our perception of virtual environments. Thus, even though most VR gear projects a roughly 90 degree field of view, our eyes are only focusing (and bringing into focus) a small portion of it at any one time. The rest of any crisp, flawlessly rendered image is completely wasted on our eyes. Foveated rendering is a process which uses the information given by eye tracking in VR games and other apps to only fully render the objects within the direct line of sight of the viewer.

Making the World a Better Place (Real and Virtual)

Though foveated rendering requires a bit more processing power to make the constant adjustment calculations, this output is more than made up for in the reduction of processing required for the full rendering of the environment. This reduction could allow game and app designers to create more complicated and detailed environments using currently available technology, instead of having to wait for improvements in the hardware to bring their visions to life.

Eye tracking also has big implications for use in many fields outside of the gaming industry. Providing accessibility to users who don’t have full use of their hands, assisting with research on all types of cognitive processes and disorders, and improving user experiences in all types of settings are all potential applications of eye tracking in virtual reality.

Who’s On Board with Onboard Eye Tracking?

Given the huge range of applications for the technology, many big name VR developers are beginning to work on integrating eye tracking into the hardware of the VR gear they are manufacturing. Oculus, sometimes considered the grandfather of VR gaming, has been working on an eye tracking implemented headset for close to a year now, and the Samsung Gear VR, which uses a smartphone as the processing component of a VR headset, already has eye tracking functionality with the use of third-party software.

Additionally, there are a number of companies which are providing eye tracking hardware or software add-ons, essentially modifying existing VR technology to allow eye tracking to be used. These companies have a range of focuses from entertainment to science, but they all require the purchase of an already existing, non-eye tracking VR gear.

The FOVE headset, which was funded on Kickstarter and already available for preorders, touts itself to be the first set of its kind with full eye tracking capability built in, both hardware and software. The FOVE retails for $599 and is compatible with 250+ already available VR titles, games and other apps.

In Conclusion

Technology is moving ever faster towards that moment when it stops being tech and starts feeling like magic. VR has the potential to give us a whole new world to explore, play, and work in, and eye tracking in VR is a big part of that puzzle. Time will tell if the implementation of this technology will progress, but if it does, the sky’s the limit for immersive entertainment and research opportunities.

Written by Dmytro Ternovyi

I'm a VR specialist, a blogger and a tech revolution believer. I have a strong belief that AR and VR are the next Big Thing to disrupt global market and change people's habits. Running a VR software development company AppReal-VR with my partners. Want to connect? Catch me on Twitter @dimulik.

Leave a Reply