Every time a new technology is introduced two things happen: people try to describe it with existing terminology and they will try to use it for existing things. It’s an important phenomenon, because it helps the general public to understand new technology and lowers the barrier to start using it.
The downside is that it can stand in the way of demonstrating the full potential of a new technology. When it comes to Virtual Reality, trying to introduce it to consumers by focusing too much on something that’s currently popular—video—might result in them not seeing the actual storytelling potential of this emerging technology.
As the co-founder of an animation studio with a passion for both technology and storytelling, I had to start exploring VR after I got my hands on my first Google Cardboard viewer in 2014. My co-founder an I got so excited that our studio was soon filled with headsets like Oculus Rift and Gear VR.
When we started doing VR demos in our studio in the beginning of 2015, we gave almost all our visitors their very first VR experience. This has changed over the last few months: more and more people say they have experienced VR before. A study published two months ago shows that 11 percent of adults in the US have experienced VR and I believe that number is growing fast as many people have or will soon have access to Google Cardboard, Gear VR or other affordable hardware. For content creators, this is amount of attention for VR is great, because a growing audience is the ultimate motivation to start creating content for it.
However, what I started to notice is that many people who say they have experienced VR, have only tried video-based VR content captured with 360° cameras. And when I was part of the VR panel of the Emerce Facebook Conference in Amsterdam recently, I noticed that my fellow panelists—from VR company Jaunt, Real Estate website Funda and TV channel VPRO—where all working on video-based VR content. Most questions and examples given in that session where also related to 360° video, which in my opinion gave the audience only a partial idea about the potential of Virtual Reality. And now that we’re entering the “VR era” with many popular headsets being released in the next six months, I see many mass media writing about VR and 360° video as if they are the same thing—which is something that worries me.
Before I write about the other—more exciting—side of VR filmmaking, let me explain what 360° video is and why I think VR experiences based on this technology—often referred to as Cinematic VR—are gaining popularity at an exponential rate:
Technically 360° videos are standard video files that contain a distorted version of a 360° — or spherical — recording. This distortion is removed when the video is played back on a device by re-projecting the video onto a computer-generated sphere. A virtual camera in the center of the sphere let’s you look around while the video plays. On a computer or smartphone, you can do this by dragging around, but when using a VR viewer like Google Cardboard, you can look around by physically moving your head.
Because of the standardisation of both the video file format and the spherical distortion used for 360° recordings, distributing this kind of VR experience has very quickly become as simple as uploading it to YouTube or Facebook. The video service added support for 360° videos in March andsupport for stereoscopic 3D a few days ago, and the social network just made the 360° video support it added in September compatible with Gear VR.
With distribution already standardised, the market for 360° cameras is exploding. There are professional grade variants such as the NEO made by Jaunt, Nokia’s OZO, and Google’s Jump Camera Rig that requires a whopping 16 GoPro cameras. But there are also budget friendly options that use less GoPros and even affordable dedicated 360° cameras like Sphericamand Bubl.
Many video production companies—and even video enthusiasts—are investing in this kind of equipment to expand their services. Sure, producing 360° video still has a lot of technical hurdles but the software to stitch together and edit videos from multiple cameras is developing extremely fast.
Because video-based VR filmmaking is quickly getting easier and more affordable, it’s very likely to become part of many consumer’s first VR experience. Fore example, the recent Google Cardboard give-away of the New York Times directly links the technology to a video-based VR documentary. And even before that, most people that have tried Google Cardboard know the Paul McCartney concert registration, because it was one of the first pieces of content in the Cardboard App that featured something people are familiar with.
The problem I have with video-based VR is that it’s only unique selling point is the ability to look around. I admit it’s nice to see people’s reaction when you tell them they actually can look around during their first time VR experience—almost everyone we gave a first-time demo kept staring in one direction initially. But I’m completely convinced the being-able-to-look-behind-you wow effect is already wearing of.
When you think about it, there aren’t many moments in life where we look around us more than a few degrees, and even then it’s often only sideways. I’ve watched a TEDx conference and an Smartphone launch in 360 degrees, and think it’s really just a because-we-technically-can novelty. I actually felt sorry for the people that are sitting right behind the camera.
But at least VR videos are 3D, right? Some 360° are indeed presented instereoscopic 3D—the principle that adds depth by presenting each eye with a slightly offset version of an image. This principle is also used to present 3D movies which, according to research, are losing popularity fast. Do they even make 3D-enabled TVs these days? (no, they don’t)
The problem with stereoscopic 3D in 360° video is two-fold: firstly, the effect only feels natural when the distance between the two cameras that capture the video for both eyes is the same as the actual distance between your eyes— the interpupillary distance or IPD. But even if you’re lucky to have an IPD that matches the stereoscopic camera setup, 3D 360° videos can still feel “off”. This is because the 3D effect is only correct at the moment where you look in the same direction as the physical cameras on the recording rig. If you’re looking in between these few sweet spots, things can get weird.
I could go on about other downsides of video-based VR, such as resolution, frame rate and file size. But let’s focus on that “other side of VR filmmaking”.
In January, Oculus — the headset maker acquired by Facebook for $2 billionin 2014 — launched an internal production company called Story Studio. They recently announced their second VR movie called Henry. As you can see in the trailer, Henry is an animation film. That’s a totally different kind of film, but not really different from computer-generated animation films likeFinding Nemo, at least at first glance. An animation can easily be distributed as a 360° video. I’ve experimented with this approach first-hand and for an experienced 3D animator it’s actually as simple as selecting the right render settings (I use Cinema 4D with V-Ray, which has a 360° option).
The thing is… Henry will not be released as a 360° video, but as an interactive experience. More specifically, it will be a Windows app specifically designed for the Oculus Rift. Much of the animation is done in a the 3D animation software Maya, but that wasn’t used to render a video. Instead, all 3D assets where imported into a program called Unreal Engine, which is typically used to develop 3D video games.
Henry is not a video game, however: there’s no objective and you don’t have to do anything other than look around to follow the little hedgehog through the story. The big difference is that Henry knows you’re doing that — he’saware of your presence. He will show you things and look straight into you eyes at certain moments. Of course, he’s programmed to do so, but I believe it makes a big difference to the feeling of presence in VR.
The format I’ve been calling video-based VR could also be called “pre-rendered” VR. If you’ve ever created a film, animation or a complex graphic on a computer you probably know that it takes a while to save your work to a non-editable format, like a Quicktime movie or a JPG image. This process is called rendering. Everything you see in video-based VR is pre-rendered meaning the complex computing work has already been done and all your device has to do is play it back. A VR movie that’s made with a video game engine, however, is rendered at the very moment you’re looking at it. This “real time-rendered” VR content has many advantages over it’s pre-rendered counterpart.
First and foremost, it can benefit from interactivity. Like in Henry, this can be used in very subtile form to achieve great effect. For instance, programming can check if you’re facing the right direction to prevent you missing important storytelling elements. Subtile visual or sound cues can then be used to draw your attention towards the right place and the story can continue.
Real time rendering can also solve all issues regarding stereoscopic 3D: the distance between the two virtual cameras can be linked to a variable IPD control on the headset. This allows the alignment of both hardware and software to the distance between your eyes. Also, the stereo effect is correct everywhere you look, because the direction your looking in always matches the virtual “camera”.
Last but not least, real time rendering allows for positional tracking. This means that not only your head movement is being tracked, but also where your are in the physical space. This can be as extreme as room-scale tracking, found on the HTC Vive system, but I discovered that it’s the tracking of the subtile movements of the head that makes the big difference when it comes to feeling “immersed” in VR. Fact is that even if we’re not turning our heads, our body constantly moves slightly. Systems like the HTC Vive, Oculus Rift and Playstation VR use external sensors that detect this kind of motion and match the virtual world accordingly. Even in its most subtile form, this allows our brains to notice motionparallax. This is the phenomenon where nearby object react different to our motion than those far away. It not only helps us to see depth, but is also something we’re so used to seeing that it’s nearly impossible for many people to feel fully immersed when it’s absent.
I hope that the differences between video-based, pre-rendered VR films and interactive, real time-rendered ones is clear at this point, but I realize you might think I’m comparing apples with oranges: a realistic subject like Paul McCartney and a computer-generated subject like Henry. But even for capturing photorealistic content, I‘m confident that 360° video recording will be replaced by 3D scanning technologies. Microsoft is currently developing technology for this called free-viewpoint video and a company called 8i can even get these kind of captures to play in-browser in real time. Regardless of it being called video, the technology allows real time 3D rendering of photorealistic moving subjects. And like the name suggests this allow (some) positional tracking while doing so.
By now you must tink I pretty much hate video-based 360° content and think Virtual Reality is doomed to fail if it becomes the de facto standard…
Well, I’m no fan of the technology, especially for VR-purposes. But I do believe it’s a necessary evil to ignite the first wave of consumer adaption of VR. The one big benefit of video is that almost every device, including smartphones, can play back video. An iPhone 5, for example, has a screen that’s just large enough to work with Google Cardboard and plays 360° videos in VR mode just fine. A 360° video can even be played back in-browser on a decent Android phone through Kolor Eyes, a service recentlyacquired by GoPro.
Henry, on the other hand, will require an Oculus Rift that hooked up to amonstrously fast gaming PC that very few people own. And according to research by Valve, co-developer of the HTC Vive headset, even gamers aren’t planning to upgrade their PCs for VR.
On top of that, mobile VR solutions like Google Cardboard and Samsung Gear VR don’t feature positional tracking, so you can’t do a whole lot more in VR than just look around. And even if they did have this feature, the current generation of mobile hardware can’t display cinema quality 3D graphics in real-time at the resolution and frame rate required for VR.
So, for most purposes, 360° video is currently the only way to easily present high quality content to the more casual VR early adopters that are willing to buy inexpensive mobile VR hardware to give it a try. I think they will be blown away by their first experience, and maybe a few after that, followed by that “is this it?” moment. I believe that in the long run video-based VR isn’t compelling enough for the the general public to go through the relatively uncomfortable experience of wearing a VR headset.
To overcome that, developers will need to make VR-specific content that is so incredibly awesome that people forget they’re in a virtual world altogether. My rule of thumb: if you can also experience VR content in any other way than VR—by dragging around a 360° video on your Facebook timeline, for example—it won’t keep consumers engaged for long.
I’m confident that VR production and presentation technologies will be developing at a rapid rate in the next few years and that it won’t take long for real time 3D to reach the level of quality and performance on mobile devices that VR requires to deliver outstanding immersive storytelling. I think that when that moment arrives, consumer VR will really take off and 360° video will be used for the practical purposes that 360° photos serve today, like real estate showcasing and virtual tours.
I can’t wait for that moment to arrive.
Thanks for reading. This was my very first post after a year without blogging (becoming a father apparently takes more time to get used to than VR…). Before that I wrote many articles about 3D printing. I’m already working on my next article here, so follow me if you want to be the first to read it.