Apple Vision Pro Video Formats Explained

With the Apple Vision Pro’s stunning display resolution (3,386 ppi) and immense processing power, it’s clear that this headset was made for immersive media consumption. You may have seen experiences like Highlining, or the Alicia Keys: Rehearsal Room and are wondering if it’s possible to achieve the same level of quality if you aren’t working directly with Apple. The answer is: of course it is! But it’s not entirely straight forward. Let’s get into it…

Spatial Video vs Immersive Video

First, let’s clarify a few terms before we move forward. What Apple calls Spatial Video is stereoscopic 3D video content within a frame. These are experiences that are captured by either the iPhone 15 or the Apple Vision Pro itself, although you can convert any 3D stereoscopic video to spatial. Spatial video is targeted more towards consumers to capture family memories, although I do see potential for brands to use it when it becomes more discoverable through social media platforms.

Everything that wraps around the viewer, or what Google and Facebook have called VR180, is simply referred to as immersive video by Apple. The immersive experiences on the Apple TV app were delivered in this format, although both types of videos are encoded in a similar way. At launch, no fully 360 immersive experiences were available, although it’s possible to watch both monoscopic and stereoscopic 360 videos within the AVP using 3rd party or custom apps.

Currently there’s no way to launch an immersive experience outside of a frame through Apple’s file system. To do that, you need to create an app or use a third-party player app. This includes any 360 or VR180 experiences, whether mono or stereo.

The Main Difference: MV-HEVC

All of Apple’s immersive videos, whether spatial or immersive, utilize MV-HEVC. This stands for Multiview High Efficiency Video Coding. You may have heard of H.265 (HEVC), MV is very similar but allows for the left/right or top/bottom stereo 3D views to be stored in separate layers rather than in 1 large container. Apple also adds metadata to define a few key parameters around the content.

Although HEVC has been around for a number of years, MV-HEVC is currently not widely supported at all by ffmpeg or editing suites like Adobe Premiere or DaVinci Resolve. Even Final Cut doesn’t have an export option for it. For now, there’s a spatial command line tool made by Mike Swanson that can take 3D videos and encode them into the multi-layered MV-HEVC container. You can also pass parameters like fov, disparity, stereo format, etc. It relies on Apple silicon, so Windows users are out of luck for now.

Some cloud based tools have been made for conversions, but seem to be limited to working only with rectilinear side-by-side 3D videos.

Apple Vision Pro’s Processing Horsepower & Resolution

Apple Vision Pro’s screens are a whopping 3660 x 3200 pixels per eye. Although they haven’t made any official claims as to the FOV of the headset, 3rd party developers claim that it looks to be around 100° horizontal. With each screen having 3,660 horizontal pixels, this would mean that a 180° immersive experience would need about 6,000 x 6,000 pixels per eye to saturate the display. Apple’s own immersive experiences have been reported at being 4320x4320 per eye at 90fps and in HDR10. There are certainly some efficiency sacrifices being made for the framerate, but perhaps there’s also diminishing returns at even higher resolutions?

360 experiences in 3D are an entirely different challenge, as many as 12,000 x 6,000 pixels per eye would be needed to saturate the display. However, the Meta Quest 3 and Quest Pro can only reliably play 5760x2880 per eye, so Apple has a unique opportunity here to quite easily deliver a better 3D360 experience than any other headset has thus far.

A Canon executive suggested that no such camera exists that could push the limits of AVP’s display, and they are right if we’re talking about a single sensor. However, there are several 6K or 8K sensor cameras on the market that can be used in a side-by-side configuration to produce stereoscopic content.

In stress testing, I’ve heard users report content all the way up to 15K in 30fps working in the Apple Vision Pro without dropping frames. This example (downsampled on DeoVR) was tested in MV-HEVC at 700mbps 7500x7500 per eye. Although it’s possible to do it, that doesn’t necessarily mean it will work at scale. There’s still the challenge of downloading or streaming, and the long-term potential to overheat or tax the device. There’s a fine line between mind-blowing experiences and annoying people with long download times.

There may be a sweet spot here where the visual quality is well beyond what we’re used to, but could still be packaged in a way that is efficient and portable. Cameras like Insta360’s Titan can shoot 10K 3D360 in 30fps, so there’s potentially a lot of content already out there.

The Brand Opportunity on Apple Vision Pro

Some brands like Gucci, J.Crew, Lowe’s and e.l.f Cosmetics have opted to make (or at least announce so far) their own custom apps. As previously stated, building your own app allows you to launch an immersive experience without relying on 3rd party players. However, that may be cost prohibitive for some.

In the short term, many other brands have made spatial video (framed 3D) content directly downloadable. In this case, they have to rely on the user to commit to downloading a file and hope they actually complete the process and watch the content. As more distribution channels become available, I think these direct downloads will be a thing of the past.

Spatial video will likely become the expendable social media content of XR. It’s quick and easy to produce, you get your engagement and you get out. For the most part, individuals and consumers will be using it. For brands, the bold move to make a long lasting impression is to opt for a fully immersive 180 or 360 experience.

Because immersive content is interchangeable, you can also get more bang for your buck by investing in content that will work for the Apple Vision Pro as well as current and future Meta Quest headsets and others. There’s no secret sauce here. You just swing for the fences on production quality, and cover your bases by future-proofing your content. I know some people are probably tired of hearing that, but everyone said it 10 years ago and look where we are now. I never would have thought we would be talking about how 8K playing on a tiny device is “just average,” but here we are.

Let's Make Something Cool!

This shift in XR display technology is every bit as significant as TV’s jump from standard definition to HD. Apple is only the first to get there. Samsung is already touting a display on their next headset that could be 4,300 x 4,300 per-eye. Meta is teaming up with LG on what might be the next generation Quest Pro. Delivering immersive experiences for these devices will no doubt be challenging, but very rewarding.

At 360 Labs, shooting ridiculously high resolution immersive video is something we've been doing for 10+ years. We have that same level of experience in immersive video post production, so the crazy ideas we come up with are backed by actual results that can be delivered. We’re excited by the possibilities opened up by the Apple Vision Pro, let’s make some amazing experiences!

References & Further Reading

• Spatial Video - Mike Swanson - a far more technical overview of what spatial and MV-HEVC is, for those seeking a deep dive.
• Apple HEVC Stereo Video - Interoperability Profile (Beta) - Apple's documentation on MV-HEVC requirements.