Spatial computing
One week after the visionOS and Vision Pro announcement, it seems clear we’re about to start a new phase of our relation with information and knowledge. Breaking free from the limitations of a flat rectangular support, new forms of representation and interactions are about to emerge. Because they will affect expectations from users, this is consequential for all digital products and the interface design practice.
These initial observations are based from what Apple has communicated so far, and first hand experience reports from people who were lucky enough to get a demo last week. We don’t have the full picture yet, but enough structural elements are before us, pointing to a place where using a mouse is going to feel like the past.
Vision Pro
All accounts place this device capabilities well above other current headsets. Together, they enable the comfortable and reliable experience testers have been describing. They make the device almost disappear and let the user focus on the content.
- Field of view: iso to natural vision.
- Image quality: pixels are not perceptible, images refresh at a 90Hz rate. This makes reading text a non issue.
- Interpretation of the environment: all tests were done in spaces specifically designed for the demos, so reports might not reflect real life use. But these demos showed very stable virtual content and very little visual distortion from reality in pass through mode, allowing for the sentiment of really “being there”, despite having no direct sight.
- Eye tracking: very accurate. Active elements must have minimum size and spacing, but targeting them by just looking at them is effortless and feels natural after a few minutes. To me, this will have major implications on perception of pointer based interfaces.
- Hand tracking: very accurate, no reports of false positives in gesture recognition. Improvements to be made in collision detection when looking closely.
- 3D pictures and videos: no one did shoot, but the 3D pictures and videos were shown.
- Outfacing screen: I have a bit of trouble with this one. The engineering effort that went into it is colossal, but the functionality it offers seems limited. When we wear sunglasses, we can’t see each others eyes, and seem to survive with it. It does give people with disabilities the possibility to be expressive to others while wearing the headset without talking or moving. I expect this space will never be at developers reach, just like watch faces on the Apple Watch.
All these hardware capabilities make it possible to anchor virtual content to your environment persuasively, and to interact with it naturally using your hands with less effort than it takes to move a mouse around on a desk. And very importantly, makes text crisp and as comfortable to read as on our screens. Given the prevalence of text in interfaces, this is major news.
visionOS
Most of the features shown during the demos and WWDC video sessions were either adaptations of existing iOS and iPadOS software, or 3D entertainment content. They nevertheless left testers with the sentiment of great quality and immense potential.
The Shared Space, where apps can co-exist as windows or volumes, uses well known patterns from iOS and macOS (windows, sidebars, tabs, buttons,…), making visionOS very consistent and immediately familiar.
Apps live in windows, volumes or spaces. Direct manipulation of 3D content with hands is possible, but the absence of tactile feedback will necessitate providing distinctive visual signals, possibly enriched with sound. Rich interaction in volumes are to me the most promising aspect of this platform, but Apple hasn’t shown any examples of it yet.
Privacy is of course part of the OS: apps can’t track the positions of the eyes, they only receive the taps and grabs intentions.
Usage
Demos have shown browsing the web, collaborating with others with Keynote and FaceTime, watching movies, viewing photos and videos (2D and 3D). They also shown the possible range of immersion, from none to complete. All these apps were in windows, frosty glass sheets distributed in the user’s space.
The race car gaming example was a bit of an absurd moment: a person immersed in a recreated 3D space was playing a game built with 3D elements rendered in a virtual 2D screen. This is not going to last.
Awkwardness and social isolation displayed in the keynote examples have been well pointed out. They are real problems, but I think our frame of reference will shift once we experience the product benefits. Many kids in the late 80s didn’t see the face of their dad too when blowing their birthday cake candle, because it was behind a massive VHS camcorder. That was accepted because it was incredible to have the birthday party playing on the TV set. 3D videos might very well be as compelling to us today and a reason enough to accept the weird goggles for a few minutes in a social event.
Demos were 30 minutes long, so we don’t have a clear indication about eye fatigue which might be an issue. If it is, I expect Apple will put warnings in place for this health subject.
The absence of richer apps existing in volumes could be explained by the hardware recent availability; it may very well be that engineers and designers haven’t spent much time with a device as capable as the one we have discovered last week. It’s possible to start building interaction patterns ported from iOS to visionOS without a fully functional device. But to design and build GarageBand or iMovie for spatial computing, in a volume, not having the combinaison of the device capabilities listed above prevents you from trying out ideas and iterate on them.
If you’re not sure about relevant use cases for productivity apps in volumes, here are some questions:
- how many architects or designers will refuse to present their work to clients using this technology and will prefer to stick to showing 3D render on a screen?
- which medical student will refuse to discover and learn the structure and functioning of the human body in volume?
- which engineering manager is going to refuse to validate a manufacturing project without inspecting its parts from every angle as if they were already built before hand?
- will there be an animation studio forcing artists to model, paint and animate 3D content on flat screens, instead of working on them in actual 3D in front of them?
Tools
Coming later this month. Namely: Xcode, Reality Composer Pro, and the visionOS Human Interface Guidelines. The fact that none of this was ready for WWDC is another tell that the hardware might have coalesced only recently. The impatience level is high to very high.
In the meantime, I want to encourage every interface designer looking forward to start building for visionOS to be curious about the langage of form and objects. Shapes, proportions and material all communicate something to observers, and spatial interface designers are going to have to be intentional about them. Coming from a product design background, it’s a subject I’m familiar with, and I will publish some advices here in the coming months. A simple exercise to start with is to compare two similar simple objects like glasses of water, screwdrivers, or door handles. Don’t pick complex objects. Start by listing the different qualities you perceive from them by looking at them or handling them, and try to figure out which aspect of the object triggers this perception for you. Dimension, thickness, shininess, precise features,…
Ressources
- First impressions: Yes, Apple Vision Pro works and yes, it’s good, by Matthew Panzarino on TechCrunch.
- Apple Vision Pro: Experiencing the Future, by Myke Hurley and CGP Grey on Cortex
- First Impressions of Vision Pro and VisionOS, by John Gruber on Daring Fireball
- Apple Vision Pro: A Watershed Moment for Personal Computing, by Federico Viticci on MacStories
- Apple Vision Pro Impressions! by Marques Brownlee
- I tried Vision Pro And Saw The Future Of Filmmaking, by Matti Haapoja
- Apple Vision Pro - A few thoughts, by Dave2D
- Design for spatial interfaces, by Apple
- Principles of spatial design, by Apple
- Design for spatial input, by Apple