Apple Vision Pro Transforms Surfaces into Touchscreens for App Interactivity

The availability of the visionOS DevKit for developers marks an exciting step toward the release of Apple Vision Pro, expected in early 2024. This spatial computer promises a host of innovative capabilities, some of which developer Steve Troughton-Smith has been sharing. One notable feature is the ability to place a vision window flat on a desk, revolutionizing the realism and user experience of Apple Vision Pro apps.

With this functionality, tasks like playing a song or typing on a virtual keyboard can now have a more authentic look and feel. Interestingly, Apple filed a patent in 2016, subsequently granted in 2020, that explores this technology employed by the Apple Vision Pro.

While the direct link to the patent may no longer be accessible, it was previously covered by AppleInsider. According to their report, the patent emphasizes that “touching objects with our hands is a natural way for humans to interact with real objects.” It goes on to mention that touch screens, which can detect and localize touches on their surfaces, have become ubiquitous in modern devices like smartphones and tablets.

This patent and the upcoming visions DevKit for developers provide a glimpse into the remarkable advancements we can anticipate from Apple Vision Pro, enhancing our interaction with technology in innovative and intuitive ways.

In exploring the possibilities of touch interaction, Apple scientists considered the option of physically equipping objects or the human body with touch-sensing sensors. However, they also recognized the potential of using cameras to track the user’s fingers, which is precisely the approach taken by Vision Pro.

By combining the insights from the patent and the demonstration provided by Troughton-Smith using vision, it becomes evident that interacting with virtual objects can be remarkably realistic when there is a physical surface involved. With Vision Pro, not only does the app remain fixed to the surface, but the interaction itself feels as natural as approaching a book resting on a table or a keyboard placed on a surface.

While we eagerly await the release of Apple Vision Pro, set to arrive in a few months, the availability of the SDK and visionOS beta ensures that we will continue to learn more about Apple’s groundbreaking spatial computer. These developments will undoubtedly shed further light on the immense potential and capabilities of this pioneering technology.