Capturing Spatial Video

In September 2023, Apple released the iPhone 15 Pro which will be capable of capturing Spatial Video for the Vision Pro using two of the lenses. This will drastically reduce the complexity of Producing Spatial Video. While the interocular distance (the distance between the two lenses) is closer than ideal, it’s quite possible that LIDAR and additional sensors can be utilized to computationally enhance 3d separation.

Capturing options

In addition to the iPhone 15 Pro, You have two options for capturing stereoscopic content: two lenses side by side or using a pane of glass as a beam-splitter to essentially do the same thing in a more compact way. Side by side is simpler, though beam-splitting is preferred by cinematographers in the film industry.

Existing formats and platforms

There are several camera/lens combos optimized for existing formats such as VR180 which may end up being a good way to start capturing stereoscopic video for the Vision Pro.

VR180

• Canon EOS VR System: a dual ultra wide fisheye lens that captures two image circles on a single 8k frame by the Canon EOS R5 or R5C. The two images on a single frame are then post-processed to VR180. It may be possible to instead use this raw footage in Producing Spatial Video. I highly recommend the R5C to avoid overheating issues while recording 8K video.
• Z CAM K1 Pro: A similar approach to the canon system but lower resolution, recording on two separate SD cards, stitched together later in software. The hardware retails for about $3k. For experimentation, renting the Canon system for a week would be cheaper.
• CALF 3D VR180 is kickstarting at $1,400 which records on two Sony CMOS sensors to a single SD card.
• Canon is showing a consumer-oriented 360/180 camera that has potential as a consumer-level device but has not officially been announced yet.

Side by side brackets

There are also several options for building side by side stereoscopic rigs:

• Andrew Lauren has created multiple stereoscopic travel rigs.
• Sunwayfoto makes a side by side camera bracket that allows for adjusting the base stereo (the inter-lens separation, or distance between the centers of each lens). Available at B&H.
• Thom Persson built a side by side stereo rig as part of his EE degree.
• Stereotec offers a few higher end side by side rigs.
• 3D Film Factory also has a side by side rig for $2,400.

Beam-splitting rigs

Most split-beam stereoscopic rigs are aimed at professional cinematographers:

• Cinegears Mini 3D Rigcosts around $31k.
• Stereotec offers several beam-splitting rigs and provided a rig used in filming Avatar 2.
• 3D Film Factory offers several beam-splitting rigs starting at $5k.
• Screenplane has several beam-splitting rigs for sale or rental.

Books

A lot of stereoscopic training material is from the last 3D boom of the early 2000’s.

• Bernard Mendiburu wrote a book called 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. It’s from 2009 so a lot of the software solutions are likely out of date, the fundamentals should still apply.
• Franz Pagot wrote Immersive 3-D: The secrets of beautiful stereo cinematography in 2014, which looks at both technical and artistic implications of the format.
• Richard Kroon wrote 3D A-to-Z which looks interesting but is hard to find and not available digitally.

Training

• 3D Film Factory offers an intense two-day workshop.
• David Bush teaches 3D production and post.

How Apple does it

I haven’t experienced the Vision Pro guided tour, but from what John Gruber and others have said, it sounds incredible:

Lastly, we saw two sports demos: an at-bat from a baseball game at Fenway Park (Phil Schiller’s hands are all over that one), and a scoring play from a Nuggets-Suns NBA basketball game. For the baseball game, the perspective wasn’t even from the stands, but rather from the home team’s dugout, ground level, right behind first base. It’s not quite just like being there, but it’s a lot like being there. It’s more realistic than seems possible. You choose where to direct your gaze: at the batter at home plate, at the pitcher, or out in the outfield. Or above the outfielders, at the scoreboard. For the NBA game, the perspective was courtside, right behind the basket. But better than the actual courtside perspective, because the perspective was slightly elevated above seating level. Fully immersive, fully three-dimensional, and seemingly perfectly to scale. Kevin Durant looked about 6’10”, right in front of me. Getting the scale just right is obviously the correct way to present this, but it seems devilishly tricky to actually pull off. Apple has pulled it off. These baseball and basketball scenes were shot by Apple using entirely custom camera rigs, and stored in altogether new file formats. This is nothing at all like 2D footage extrapolated into 3D, or just painted on a virtual circular wall around you. It looks real. It seems as profoundly different from watching regular TV telecasts of sports as TV telecasts are from audio-only radio broadcasts.2 It was incredible. I would genuinely consider buying a Vision Pro if the one and only thing it did was show entire sporting events like this.

Back in 2020 Apple bought a company called NextVR that specializes in stereoscopic capture of sporting and other events. At the time of purchase, they were focused on VR, but I’m certain this team and tech have been deployed to support video on the Vision Pro. Based on images and coverage from before the purchase they were at least using side by side rigs with extreme wide angle lenses on high resolution RED cinema cameras. Front view of a NextVR rig from The Verge. Back view of a NextVR rig from The Verge.