Apple visionOS Tech Stack
SwiftUI / SwiftData / RealityKit / ARKit / Reality Composer Pro
Swift · SwiftUI · RealityKit · RealityView · ARKit · Reality Composer Pro · MaterialX · USDZ · SwiftData · Swift Charts · Swift Structured Concurrency · Swift Package Manager · XCTest · other modern frameworks.
Summery of how everything is related!
SwiftUI is a declarative framework for building user interfaces. It provides a modern and expressive way to create beautiful and functional UIs.
SwiftData is a data management framework that provides a unified API for accessing and manipulating data from a variety of sources, including local databases, remote servers, and cloud storage.
RealityKit is a framework for creating 3D augmented reality (AR) experiences. It provides a number of features that make it easy to create AR apps, including scene tracking, object detection and tracking, and image tracking. It also provides a number of features for blending virtual content with the real world, such as anchors and portals. It is also used to add spacial audio (AudioComponent), and it’s used for handle interactions with the entities. RealityKit can be used to create 3D content for both windows and volumes, but it is most commonly used for immersive spaces. RealityView is a SwiftUI view that allows you to display RealityKit content in a SwiftUI app.
Note: You can use Xcode with Reality Composer Pro to build a scene with spatial audio (with PHASE — Physical Audio Spatialization Engine)
ARKit is a framework that provides a number of features that make it easy to develop AR apps, including scene tracking, object detection and tracking, image tracking, plane detection, and hand tracking.
NOTE: Both RealityKit and ARKit provide scene tracking, object detection and tracking, and image tracking. RealityKit is a higher-level framework that builds on top of ARKit. ARKit is a lower-level framework, more flexible than RealityKit, but it requires more code to use.
Reality Composer Pro is a powerful tool for creating 3D content for visionOS apps. It provides a number of features that make it easy to create and manage 3D scenes, including a visual editor, a node-based material editor, and support for USDZ, a universal file format for 3D content.
MaterialX is an open standard for describing materials in a physically based way. It is used in a variety of industries, including film and television, video games, and architecture. MaterialX is used in Reality Composer Pro and visionOS to create realistic and believable materials for 3D objects.
USDZ is a file format for storing 3D assets that was developed by Apple in collaboration with Pixar. It is based on the Universal Scene Description (USD) format, which is a powerful and flexible system for managing complex 3D scenes and assets. USDZ files are compressed and can contain all of the necessary information to display a 3D object or scene without any external dependencies. This makes them ideal for sharing 3D assets on the web and in mobile apps.
All of these components work together to provide a powerful and versatile platform for developing visionOS apps. By using these different components, developers can create immersive and engaging experiences for their users.
Example
A visionOS developer might use SwiftUI to create a UI for their AR app, SwiftData to manage data from a remote server, RealityKit to create the AR experience itself, and Reality Composer Pro to create the 3D content. The developer might also use MaterialX to create realistic materials for their 3D objects, and USDZ to share their 3D assets with other developers.
Below are the developer tools …
Developer Tools
XCode
The following are some new features and improvements in visionOS 15.1 beta for visionOS programming:
RealityKit
New APIs for spatial audio: These APIs allow developers to create more immersive and realistic AR experiences by controlling the spatial location of audio sources.
New features for creating and managing 3D scenes: These features make it easier to create and manage complex 3D scenes, including new tools for importing and exporting scenes, and new ways to manage scene hierarchy and lighting.
ARKit
Improved object detection and tracking: ARKit has been improved to better detect and track objects in the real world, including smaller and more complex objects.
New APIs for hand tracking: These APIs allow developers to create AR experiences that interact with the user’s hands.
Improved support for multiple devices: ARKit has been improved to support multiple devices working together to create shared AR experiences.
Reality Composer Pro
New tools for creating spatial audio: Reality Composer Pro now includes tools for creating and editing spatial audio, making it easier to add spatial audio to AR experiences.
Improved support for USDZ: Reality Composer Pro now supports more features of the USDZ file format, making it easier to import and export 3D assets from other applications.
New features for creating and managing 3D materials: Reality Composer Pro now includes new tools for creating and managing 3D materials, including a new node-based material editor.
Reality Composer Pro (MaterialX)
Is a powerful tool for creating 3D content for visionOS apps. It provides a number of features that make it easy to create and manage 3D scenes, including:
- A visual editor that allows you to create and arrange 3D objects, materials, and lighting.
- A node-based material editor that allows you to create complex materials and effects.
- Support for USDZ, a universal file format for 3D content.
- Tight integration with Xcode, the IDE for developing visionOS apps.
Reality Composer Pro provides a node-based material editor that makes it easy to create and edit MaterialX materials.
- Reality Composer Pro provides a node-based material editor that makes it easy to create and edit MaterialX materials.
- Reality Composer Pro provides a number of pre-built MaterialX materials that you can use in your projects.
- Reality Composer Pro is tightly integrated with Xcode, the IDE for developing visionOS apps, making it easy to import and use MaterialX materials in your visionOS apps.
MaterialX
Reality Composer Pro provides a node-based material editor that makes it easy to create and edit MaterialX materials. Reality Composer Pro also provides a number of pre-built MaterialX materials that you can use in your projects.
If you are new to MaterialX, I recommend using it with Reality Composer Pro. Reality Composer Pro provides a number of features that make it easier to learn and use MaterialX.
Here are some of the benefits of using Reality Composer Pro with MaterialX:
- Reality Composer Pro provides a node-based material editor that makes it easy to create and edit MaterialX materials.
- Reality Composer Pro provides a number of pre-built MaterialX materials that you can use in your projects.
- Reality Composer Pro is tightly integrated with Xcode, the IDE for developing visionOS apps, making it easy to import and use MaterialX materials in your visionOS apps.
Here is all our articles on Medium:
Resources from Apple:
More resources coming soon …
Summery of Important Concepts
How SwiftUI and RealityKit (RealityView) fit into visionOS.
- SwiftUI is a modern user interface framework that makes it easy to create beautiful and responsive UIs. SwiftUI is declarative, which means that you describe what you want your UI to look like, and SwiftUI takes care of the rest.
- RealityKit is a framework for creating 3D and augmented reality experiences. RealityKit makes it easy to create and manipulate 3D objects, and to interact with them in real time.
- RealityView is a SwiftUI view that allows you to display 3D content in your app. RealityView uses RealityKit to render the 3D content, and allows you to interact with it using SwiftUI gestures.
- Together, SwiftUI and RealityKit provide a powerful and flexible platform for developing immersive and interactive applications for VisionOS.
Understand 3D apps with visionOS
- ARKit is a framework for developing augmented reality experiences. ARKit can be used to track the user’s environment and to place 3D objects in the real world.
- Reality Composer Pro to build 3D models. Use Reality Composer to create and preview your 3D content. RealityComposer is a powerful tool that makes it easy to create and edit 3D objects, materials, and scenes.
- Use iOS image capture to build 3D models
Controls and Material in visionOS
Controls and Material in VisionOS are two important aspects of the platform that allow developers to create immersive and interactive applications.
- Controls are user interface elements that allow users to interact with applications. VisionOS provides a wide range of built-in controls, such as buttons, sliders, and text fields. Developers can also create their own custom controls using SwiftUI.
- Material in VisionOS is a concept that describes how visual content is displayed on the screen. Material can be used to create a variety of effects, such as translucency, blurring, and shadows.
VisionOS provides a number of features that make it easy to integrate controls and materials into applications.
Here are some examples of how controls and materials can be used in VisionOS applications:
- Create a translucent button: This would allow the underlying visual content to be seen through the button.
- Create a blurred background: This would help to focus attention on the foreground content.
- Create a shadow around a control: This would help to make the control stand out from the background.
Spatial interactions with RealityKit and ARKit
Similarities:
- Both RealityKit and ARKit can be used to create spatial interactions, which allow users to interact with virtual objects in the real world using their physical bodies.
- Both RealityKit and ARKit use the user’s device camera to track the user’s environment and to place virtual objects in the real world.
- Both RealityKit and ARKit provide a number of features that make it easy to create spatial interactions, such as collision detection and physics simulation.
Differences:
- RealityKit is a higher-level framework than ARKit, and it provides a number of features that make it easier to create visually appealing and performant spatial interactions. For example, RealityKit provides a number of built-in materials and textures, and it has a number of features that make it easy to create complex 3D scenes.
- ARKit is a lower-level framework than RealityKit, and it gives developers more control over the spatial interaction experience. For example, ARKit provides developers with access to the raw camera data, and it allows developers to write their own custom collision detection and physics simulation code.
How to create immersive 3D content
Choose the right tools and technologies. You can use a variety of tools and technologies to create 3D content, such as: Blender, Maya, Cinema 4D, Unreal Engine, Unity.
- Reality Composer Pro
Object Capture using iOS 17 (needs LiDAR)
New feature that allows you to create high-quality 3D models of real-world objects using your iPhone or iPad. This feature is powered by LiDAR, which is a sensor that measures the distance to objects. To use Object Capture, you simply need to walk around the object that you want to scan. As you walk, your iPhone or iPad will use LiDAR to create a 3D point cloud of the object. Once the scan is complete, you can use the Reality Composer app to edit and finalize the 3D model.
Design your 3D scene. Once you have chosen your tools and technologies, you can start designing your 3D scene. This includes creating 3D models, textures, lighting, sound and animation.
Summery of Full Tech Stack
Swift: A modern, expressive programming language tailored for creating user interfaces and app logic. Its declarative nature simplifies development and ensures code readability.
Swift structured concurrency: An efficient method for managing multiple tasks simultaneously, optimizing performance and resource utilization. It enables developers to handle complex operations seamlessly.
SwiftData: A data management framework that provides a unified API for accessing and manipulating data from various sources, including local databases, remote servers, and cloud storage. It simplifies data handling and facilitates seamless data integration.
RealityKit: A powerful framework for creating immersive 3D augmented reality (AR) experiences. It enables the creation of interactive and engaging AR worlds that seamlessly blend virtual elements with the real world.
RealityView: A SwiftUI view that seamlessly integrates RealityKit content into SwiftUI apps. It bridges the gap between RealityKit’s 3D rendering capabilities and SwiftUI’s user interface design, creating a cohesive experience.
ARKit: A framework for developing AR experiences that leverages the device’s sensors to track the user’s environment and place virtual objects realistically in the real world. It provides advanced spatial awareness capabilities for creating truly immersive AR experiences.
MaterialX: An open standard for describing materials. It provides a consistent and flexible way to define material properties, enabling developers to create realistic and visually appealing 3D objects.
Reality Composer Pro (MaterialX): A powerful tool for creating 3D content for VisionOS apps. It provides a visual editor for designing and editing 3D models, materials, and scenes, and also integrates with MaterialX for advanced material editing.
USDZ: A file format for storing and sharing 3D assets. It is a compact and efficient format that can be used to share 3D assets across various platforms, including VisionOS and other AR/VR applications.
Xcode: An integrated development environment (IDE) specifically designed for building VisionOS applications. It provides a comprehensive set of tools and features for developing, debugging, and testing AR/VR applications.
These components collectively provide developers with the necessary tools and frameworks to create innovative and immersive augmented reality experiences on VisionOS.
~Ash
