Envision Building Magic with visionOS
VisionDevCamp — Building the Future, Today :-)
Here are some resources and strategies to use when building at the visionOS Hackathon.
Apple Sample visionOS Code
Description & link to all Apple visionOS sample code.
Reuse as much of this code in your hack-a-thon as possible.
Picasso “good artists copy, great artists steal”
Introductions (about me :-)
Way too much info about me
Reuse Swift
When building try reuse your Swift knowledge as much as possible.
Swift · SwiftUI · SF Symbols · SwiftData · Swift Structured Concurrency · Combine · Swift Package Manager · XCTest · CI/CD (GitHub Actions)
SwiftData
SwiftData makes data management ridiculously easy and integrates into SwiftUI!
Simple steps to use SwiftData:
With only a few lines of code the entire persistence layer is done and it works with SwiftUI (reactive).
Resources:
Project:
SF Symbols
The entire UI is all done with nothing but SF Symbols.
SwiftCharts
Super easy to build chats and bind them to SwiftUI.
- define struct
- Fill with data or use SwiftData
- Tell SwiftChart the type of chart you want
import SwiftUI
import Charts
struct ToyShape: Identifiable {
var color: String
var type: String
var count: Double
var id = UUID()
}
var stackedBarData: [ToyShape] = [
.init(color: "Green", type: "Cube", count: 2),
.init(color: "Green", type: "Sphere", count: 0),
.init(color: "Purple", type: "Cube", count: 1),
.init(color: "Purple", type: "Sphere", count: 1),
.init(color: "Pink", type: "Cube", count: 1),
.init(color: "Pink", type: "Sphere", count: 2),
.init(color: "Yellow", type: "Cube", count: 1),
.init(color: "Yellow", type: "Sphere", count: 1),
]
struct ChartsUIView: View {
var body: some View {
Chart {
ForEach(stackedBarData) { shape in
BarMark(
x: .value("Shape Type", shape.type),
y: .value("Total Count", shape.count)
)
.foregroundStyle(by: .value("Shape Color", shape.color))
}
}
}
}
#Preview {ChartsUIView()}
SwiftUI
A simple window look amazing in visionOS
- TabView holds the Ornaments on the right side and
SaleButtonStyleis the red “Sale” sign.
import SwiftUI
import SwiftData
@main
struct FireCarApp: App {
let modelContainer: ModelContainer
init() {
do {
modelContainer = try ModelContainer(for: ShoppingCart.self)
} catch {
fatalError("Could not initialize ModelContainer")
}
}
var body: some Scene {
WindowGroup (id: "MainWindow") {
TabView {
ContentView()
.tag(1) // Tag for the first tab
.tabItem {
Label("Home", systemImage: "house")
}
.ornament(attachmentAnchor: .scene(.topTrailing) ,contentAlignment: .center) {
Label("Sale", systemImage: "storefront")
.font(.system(size: 27, weight: .bold)) // Scale font size based on width
.foregroundColor(.white)
.padding()
.background(Color.red)
.cornerRadius(10)
.shadow(radius: 5)
.buttonStyle(SaleButtonStyle())
}
ShoppingCartExampleView()
.tag(2) // Tag for the second tab
.tabItem {
Label("Cart", systemImage: "cart")
}
.ornament(attachmentAnchor: .scene(.bottomTrailing) ,contentAlignment: .center) {
Label("Sale", systemImage: "storefront")
.font(.system(size: 27, weight: .bold)) // Scale font size based on width
.foregroundColor(.white)
.padding()
.background(Color.red)
.cornerRadius(10)
.shadow(radius: 5)
.buttonStyle(SaleButtonStyle())
}
ChartsUIView()
.tag(3) // Tag for the second tab
.tabItem {
Label("Charts", systemImage: "chart.bar")
}
}
}.modelContainer(modelContainer)
WindowGroup ("Shopping Cart", id: "SecondWindow"){
HStack {
//ShoppingCartUIView()
ShoppingCartExampleView()
}
}.modelContainer(modelContainer)
ImmersiveSpace(id: "ImmersiveSpace") {
ImmersiveView()
}
}
}
struct SaleButtonStyle: ButtonStyle {
func makeBody(configuration: Configuration) -> some View {
configuration.label
.padding()
.background(.regularMaterial, in: .rect(cornerRadius: 12))
.hoverEffect()
.scaleEffect(configuration.isPressed ? 0.95 : 1)
}
}
Normal SwiftUI looks great in visionOS.
Swift Structured Concurrency
Please reuse as much of Swift as possible in your visionOS projects.
On top of the Swift Stack we build our visionOS App using AR/VR.
visionOS Stack
RealityKit · RealityView · ARKit · Reality Composer Pro · MaterialX · USDZ
Summery of how everything is related!
RealityKit (RealityView)
NOTE: Both RealityKit and ARKit provide scene tracking, object detection and tracking, and image tracking. RealityKit is a higher-level framework that builds on top of ARKit. ARKit is a lower-level framework, more flexible than RealityKit, but it requires more code to use.
The SwiftUI view FireCarApp calls ImmersiveView which starts the RealityKit and it loads the Immersive scene into the RealityView which is a SwiftUI view.
In the packages (on the left) you can see the RealityKitContent with the Immersive scene (above the Materials folder). If we click on the Immersive scene we can load the scene in Reality Composer Pro.
Reality Composer Pro
Reality Composer Pro: A Tool for Building 3D and AR Experiences
What it is: A developer tool for designing, editing, and previewing 3D content for use in augmented reality (AR) applications.
What it does:
- Imports and organizes 3D models, materials, and sounds.
- Creates RealityKitContentBundle which combines all assets for Xcode projects.
- Integrates with Xcode for a streamlined workflow.
Key Concepts:
Entity Component System (ECS): A foundational structure for managing 3D content:
- Entities: Basic building blocks representing objects in the scene.
- Components: Define attributes and behaviors of entities (e.g., position, physics).
- Systems: Control interactions and updates between entities and their components.
Custom Components: Allow extending entity functionality beyond pre-defined ones.
Attachments API: Enables embedding interactive SwiftUI views within AR scenes for UI elements.
Dynamic Content Generation: Creating and modifying AR scene elements programmatically based on user interactions or other factors.
Overall, Reality Composer Pro empowers developers to create interactive and adaptable 3D and AR experiences.
The below article is a guide to Reality Composer Pro, a tool for building AR experiences. It covers loading 3D content, the Entity Component System (ECS) for structuring your AR scenes, creating custom components, integrating UI elements using the Attachments API, and generating dynamic content. It also showcases Reality Composer Pro with the Apple Diorama App.
- Reality Composer Pro is a tool for creating 3D and augmented reality (AR) experiences.
- It allows you to import and organize 3D models, materials, and sounds.
- Reality Composer Pro creates a RealityKitContentBundle which includes everything needed to use the 3D content in your Xcode project.
- RealityKit uses the Entity Component System (ECS) for managing 3D content.
- Entities are the basic building blocks, components define attributes and behaviors, and systems control interactions and updates.
import Foundation
// Component: Represents a specific aspect of an entity
struct PositionComponent {
var x: Float
var y: Float
}
struct HealthComponent {
var healthPoints: Int
}
// Entity: Represents an object in the game world
struct Entity {
var id: Int
var position: PositionComponent
var health: HealthComponent
}
// System: Performs operations on entities with specific components
struct MovementSystem {
func move(entity: inout Entity, deltaX: Float, deltaY: Float) {
entity.position.x += deltaX
entity.position.y += deltaY
print("Entity \(entity.id) moved to (\(entity.position.x), \(entity.position.y))")
}
}
struct HealthSystem {
func damage(entity: inout Entity, amount: Int) {
entity.health.healthPoints -= amount
print("Entity \(entity.id) took \(amount) damage. Health: \(entity.health.healthPoints)")
}
}
// Creating entities and systems
var player = Entity(id: 1, position: PositionComponent(x: 0, y: 0), health: HealthComponent(healthPoints: 100))
var enemy = Entity(id: 2, position: PositionComponent(x: 10, y: 5), health: HealthComponent(healthPoints: 50))
let movementSystem = MovementSystem()
let healthSystem = HealthSystem()
// Performing actions using systems
movementSystem.move(entity: &player, deltaX: 2, deltaY: 3)
movementSystem.move(entity: &enemy, deltaX: -1, deltaY: 0)
healthSystem.damage(entity: &player, amount: 20)
healthSystem.damage(entity: &enemy, amount: 10)- Reality Composer Pro allows you to create custom components to extend the functionality of entities beyond the predefined components provided by the framework.
- The Attachments API allows you to embed SwiftUI views into your 3D AR scenes. This enables you to create interactive UI elements within your AR experience.
- Dynamic content generation refers to the process of creating and modifying elements within an AR scene programmatically. This concept is essential for creating AR experiences that adapt and respond based on user interactions or other factors.
Omniverse USD Composer vs. Reality Composer Pro
Omniverse USD Composer offers a powerful and highly customizable toolchain, ideal for professional 3D artists working with complex scenes and requiring interoperability across platforms. Reality Composer Pro prioritizes ease of use and real-time performance for AR development within the Apple ecosystem. Its streamlined toolchain might not be suitable for highly detailed 3D assets but excels in rapid prototyping and iteration for AR applications. Both use USD(z) as their foundation to build on.
MaterialX
On the bottom of the Reality Composer Pro you will see the MaterialX Shader Graph.
- Aims to provide a common way to represent material definitions across different software and rendering engines.
- Benefits include portability, collaboration, and version tracking of materials.
- Uses node graphs for flexibility and customization.
- Components include shaders, patterns, and surfaces.
Reality Composer Pro
- Uses MaterialX as the foundation for creating custom materials.
- Offers ShaderGraph editor, a visual tool for building materials using nodes.
- Supports two main shader types: Physically Based (PBR) and Custom.
- PBR simulates real-world material properties for realistic appearances.
- Custom shaders offer precise control over object appearance, including animations and effects.
Building Custom Materials
- Done in the ShaderGraph editor by connecting nodes.
- Nodes can represent color manipulation, texture sampling, and more.
- Users can adjust node properties and preview the material in real-time.
- Materials can be reused across different objects in a project.
Additional Concepts
Node Graphs: Visual representations of workflows or systems using connected nodes.
Custom Nodes: User-defined nodes for encapsulating complex functionality.
Geometry Modifiers: Components that dynamically alter the geometry of 3D objects.
Please make sure to grab all the resources (models, materials, ect … fireworks, clouds, sounds, maps) from the Apple sample code to use in your projects.
What each source example focuses on …
USDZ
Here’s a quick overview of OpenUSD:
OpenUSD (USD) is an open-source framework for describing, composing, simulating, and collaborating on 3D worlds.
Originally developed by Pixar, it’s now an industry standard for 3D interchange between different software programs used in visual effects, animation, and other fields like architecture and robotics.
Think of it as a common language for 3D data, allowing seamless exchange of models, textures, materials, and animations across different tools.
Key benefits:
- Collaboration: Enables teams using various software to work on the same 3D scene efficiently.
- Interoperability: Avoids the hassle of data conversion between different 3D applications.
- Version control: Tracks changes and versions of 3D assets for easier management.
OpenUSD uses a scene description approach, where data is organized in a hierarchical structure with elements like geometry, shading, lighting, and physics represented separately.
It leverages node graphs for flexibility and customization, allowing users to build complex scenes by connecting different nodes.
Tools like Pixar’s USD Preview and NVIDIA Omniverse USD are available for working with OpenUSD files.
Overall, OpenUSD provides a powerful and flexible foundation for creating, sharing, and modifying 3D content across various software applications.
Notes on OpenUSD vs USDZ:
ARKit
Developed by Apple to create AR experiences on iPhones and iPads. Overlays digital information onto the real world in real-time.
Key features:
- Motion Tracking: Tracks device movement for placing virtual objects.
- Environmental Understanding: Detects surfaces like floors and tables.
- Lighting Estimation: Estimates lighting for realistic shadows and reflections.
- Face Tracking: Tracks facial expressions for adding effects to faces.
- Image and Object Recognition: Triggers AR experiences based on visual markers.
- Integration with SceneKit and Metal: Provides high-quality 3D rendering.
- Multiuser AR: Allows multiple users to interact with virtual objects.
- Persistent AR: Saves and reloads AR experiences.
- LiDAR Support: Enhances AR experiences with more accurate depth sensing.
Review RealityKit
A framework by Apple to simplify AR development on Apple devices.
Review Key features:
- Easy AR Development: Creates AR applications without deep expertise in graphics or 3D programming.
- Entity-Component System: Manages virtual objects with easy customization.
- Realistic Rendering: Applies advanced lighting and shading for realistic visuals.
- Spatial Audio: Creates immersive experiences with sounds from specific virtual positions.
- Swift Integration: Works seamlessly with Swift for clean and expressive code.
- Animations and Physics: Enables dynamic and interactive virtual objects.
- AR Interaction: Allows user interaction with virtual content through gestures and taps.
- SwiftUI Integration: Combines AR experiences with native app UIs.
- AR Quick Look: Lets users preview 3D models directly in AR.
- Xcode Integration: Provides a comprehensive toolkit for building AR applications.
RealityKit & ARKit
Work together to create AR experiences.
- ARKit: Tracks real-world features and provides spatial context.
- RealityKit: Renders and interacts with virtual objects in the AR environment.
Collaboration between ARKit and RealityKit:
- AR Scene Management: ARKit tracks the device and detects features. RealityKit uses this information to place virtual entities.
- Entity-Component System: RealityKit creates and manages virtual entities based on ARKit data.
- Visual Rendering: RealityKit renders virtual entities using lighting and shading information from ARKit.
- Interaction and Animation: ARKit provides user interaction data. RealityKit uses this data to trigger animations and interactions with virtual entities.
- Spatial Audio: Both ARKit and RealityKit support spatial audio.
RealityView in visionOS
- A component in visionOS to integrate RealityKit content into your app.
- Uses RealityViewContentProtocol to manipulate 3D entities.
- RealityView is asynchronous to avoid app hangs while loading content.
- RealityView displays 3D content in true 3D space within your app.
Putting it all together
In visionOS, RealityKit, ARKit, and RealityView work together for AR development:
- RealityKit builds 3D content.
- ARKit understands the real-world context.
- RealityView enables 3D interactions with SwiftUI.
Here’s how these components interact:
- RealityKit is used to design, customize, and interact with 3D content.
- ARKit understands the real world environment.
- RealityView embeds RealityKit content into the SwiftUI interface.
- You can define interactions and behaviors for 3D content using RealityKit.
- ARKit provides data to position and anchor RealityKit content in the real world.
RealityView acts as a bridge between RealityKit and ARKit. It allows you to create AR experiences where virtual content interacts with the real world.
Hand Tracking
Only ARKit does hand tracking and it is a bunch of messy code. Use this Apple sample app to look at how hand tracking is done.
When you see crazy code like this …
// Get the position of all joints in world coordinates.
let originFromLeftHandThumbKnuckleTransform = matrix_multiply(
leftHandAnchor.originFromAnchorTransform, leftHandThumbKnuckle.anchorFromJointTransform
).columns.3.xyz
let originFromLeftHandThumbTipTransform = matrix_multiply(
leftHandAnchor.originFromAnchorTransform, leftHandThumbTipPosition.anchorFromJointTransform
).columns.3.xyz
let originFromLeftHandIndexFingerTipTransform = matrix_multiply(
leftHandAnchor.originFromAnchorTransform, leftHandIndexFingerTip.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandThumbKnuckleTransform = matrix_multiply(
rightHandAnchor.originFromAnchorTransform, rightHandThumbKnuckle.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandThumbTipTransform = matrix_multiply(
rightHandAnchor.originFromAnchorTransform, rightHandThumbTipPosition.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandIndexFingerTipTransform = matrix_multiply(
rightHandAnchor.originFromAnchorTransform, rightHandIndexFingerTip.anchorFromJointTransform
).columns.3.xyz
let indexFingersDistance = distance(originFromLeftHandIndexFingerTipTransform, originFromRightHandIndexFingerTipTransform)
let thumbsDistance = distance(originFromLeftHandThumbTipTransform, originFromRightHandThumbTipTransform)
// Heart gesture detection is true when the distance between the index finger tips centers
// and the distance between the thumb tip centers is each less than four centimeters.
let isHeartShapeGesture = indexFingersDistance < 0.04 && thumbsDistance < 0.04
if !isHeartShapeGesture {
return nil
}
// Compute a position in the middle of the heart gesture.
let halfway = (originFromRightHandIndexFingerTipTransform - originFromLeftHandThumbTipTransform) / 2
let heartMidpoint = originFromRightHandIndexFingerTipTransform - halfway
// Compute the vector from left thumb knuckle to right thumb knuckle and normalize (X axis).
let xAxis = normalize(originFromRightHandThumbKnuckleTransform - originFromLeftHandThumbKnuckleTransform)
// Compute the vector from right thumb tip to right index finger tip and normalize (Y axis).
let yAxis = normalize(originFromRightHandIndexFingerTipTransform - originFromRightHandThumbTipTransform)
let zAxis = normalize(cross(xAxis, yAxis))
// Create the final transform for the heart gesture from the three axes and midpoint vector.
let heartMidpointWorldTransform = simd_matrix(
SIMD4(xAxis.x, xAxis.y, xAxis.z, 0),
SIMD4(yAxis.x, yAxis.y, yAxis.z, 0),
SIMD4(zAxis.x, zAxis.y, zAxis.z, 0),
SIMD4(heartMidpoint.x, heartMidpoint.y, heartMidpoint.z, 1)
)
return heartMidpointWorldTransformYour looking at hand tracking code … and it will not run in the simulator.
Apple will make this easier over time. For now replace the hand tracking with buttons … move right/left/up/down/click buttons.
Tools
- Reality Composer Pro: Creates and previews 3D content for visionOS apps.
- Reality Composer for iOS/iPadOS: Builds, tests, and simulates AR experiences.
- Reality Converter: Converts, views, and customizes USDZ 3D objects on Mac.
- Object Capture
Make models with your iPhone/iPad
- RoomPlan
Introducing RoomPlan — Powered by ARKit, RoomPlan is a new Swift API that utilizes the camera and LiDAR Scanner on iPhone and iPad to create a 3D floor plan of a room, including key characteristics such as dimensions and types of furniture.
Thanks for coming to the talk and good luck at the hackathon!
List of articles:
This article covers everthing about visionOS programming but is almost impossible to read … we will be breaking it up soon …
~Ash
