visionOS Dev Camp Hackathon

Resources and Strategies

Vision Dev Camp Hackathon

Envision Building Magic with visionOS

VisionDevCamp — Building the Future, Today :-)

VisionDevCamp - A developer event for Apple Vision Pro and visionOS

Here are some resources and strategies to use when building at the visionOS Hackathon.

Apple Sample visionOS Code

Description & link to all Apple visionOS sample code.

zoeware - Apple Sample Code

Reuse as much of this code in your hack-a-thon as possible.

GitHub - IvanCampos/visionOS-examples: Spatial Computing Accelerators

Picasso “good artists copy, great artists steal”

Introductions (about me :-)

Way too much info about me

Siamak (Ash) Ashrafi's Speaker Profile

Reuse Swift

When building try reuse your Swift knowledge as much as possible.

Swift · SwiftUI · SF Symbols · SwiftData · Swift Structured Concurrency · Combine · Swift Package Manager · XCTest · CI/CD (GitHub Actions)

SwiftData

SwiftData makes data management ridiculously easy and integrates into SwiftUI!

The visionOS cart is powered by SwiftData

Simple steps to use SwiftData:

With only a few lines of code the entire persistence layer is done and it works with SwiftUI (reactive).

SwiftData

Resources:

SwiftData - Xcode - Apple Developer

Project:

https://www.hackingwithswift.com/quick-start/swiftdata/swiftdata-tutorial-building-a-complete-project

SF Symbols

The entire UI is all done with nothing but SF Symbols.

SF Symbols - Apple Developer

SwiftCharts

Super easy to build chats and bind them to SwiftUI.

1. define struct
2. Fill with data or use SwiftData
3. Tell SwiftChart the type of chart you want

import SwiftUI
import Charts
struct ToyShape: Identifiable {
    var color: String
    var type: String
    var count: Double
    var id = UUID()
}
var stackedBarData: [ToyShape] = [
    .init(color: "Green", type: "Cube", count: 2),
    .init(color: "Green", type: "Sphere", count: 0),
    .init(color: "Purple", type: "Cube", count: 1),
    .init(color: "Purple", type: "Sphere", count: 1),
    .init(color: "Pink", type: "Cube", count: 1),
    .init(color: "Pink", type: "Sphere", count: 2),
    .init(color: "Yellow", type: "Cube", count: 1),
    .init(color: "Yellow", type: "Sphere", count: 1),
]
struct ChartsUIView: View {
    var body: some View {
        Chart {
            ForEach(stackedBarData) { shape in
                BarMark(
                    x: .value("Shape Type", shape.type),
                    y: .value("Total Count", shape.count)
                )
                .foregroundStyle(by: .value("Shape Color", shape.color))
            }
        }
    }
}
#Preview {ChartsUIView()}

Creating a chart using Swift Charts | Apple Developer Documentation

SwiftCarts running in visionOS

SwiftUI

A simple window look amazing in visionOS

1. TabView holds the Ornaments on the right side and SaleButtonStyleis the red “Sale” sign.

import SwiftUI
import SwiftData

@main
struct FireCarApp: App {
    
    
    let modelContainer: ModelContainer
        
    init() {
        do {
            modelContainer = try ModelContainer(for: ShoppingCart.self)
        } catch {
            fatalError("Could not initialize ModelContainer")
        }
    }
    var body: some Scene {
        WindowGroup (id: "MainWindow") {
            TabView {
                ContentView()
                    .tag(1) // Tag for the first tab
                    .tabItem {
                        Label("Home", systemImage: "house")
                    }
                    .ornament(attachmentAnchor: .scene(.topTrailing) ,contentAlignment: .center) {
                        Label("Sale", systemImage: "storefront")
                                  .font(.system(size: 27, weight: .bold)) // Scale font size based on width
                                  .foregroundColor(.white)
                                  .padding()
                                  .background(Color.red)
                                  .cornerRadius(10)
                                  .shadow(radius: 5)
                                  .buttonStyle(SaleButtonStyle())
                    }
                ShoppingCartExampleView()
                    .tag(2) // Tag for the second tab
                    .tabItem {
                        Label("Cart", systemImage: "cart")
                    }
                    .ornament(attachmentAnchor: .scene(.bottomTrailing) ,contentAlignment: .center) {
                        Label("Sale", systemImage: "storefront")
                                  .font(.system(size: 27, weight: .bold)) // Scale font size based on width
                                  .foregroundColor(.white)
                                  .padding()
                                  .background(Color.red)
                                  .cornerRadius(10)
                                  .shadow(radius: 5)
                                  .buttonStyle(SaleButtonStyle())
                    }
                    
                ChartsUIView()
                    .tag(3) // Tag for the second tab
                    .tabItem {
                        Label("Charts", systemImage: "chart.bar")
                    }
                    
            }
            
        }.modelContainer(modelContainer)
           
        
        WindowGroup ("Shopping Cart", id: "SecondWindow"){
            HStack {
                //ShoppingCartUIView()
                ShoppingCartExampleView()
            }
        }.modelContainer(modelContainer)


        ImmersiveSpace(id: "ImmersiveSpace") {
            ImmersiveView()
        }
        
       
    }
}

struct SaleButtonStyle: ButtonStyle {
    func makeBody(configuration: Configuration) -> some View {
        configuration.label
            .padding()
            .background(.regularMaterial, in: .rect(cornerRadius: 12))
            .hoverEffect()
            .scaleEffect(configuration.isPressed ? 0.95 : 1)
    }
}

SwiftUI window in visionOS

Normal SwiftUI looks great in visionOS.

Swift Structured Concurrency

Swift Structured Concurrency

Please reuse as much of Swift as possible in your visionOS projects.

On top of the Swift Stack we build our visionOS App using AR/VR.

visionOS Stack

RealityKit · RealityView · ARKit · Reality Composer Pro · MaterialX · USDZ

Summery of how everything is related!

Apple visionOS Tech Stack

RealityKit (RealityView)

NOTE: Both RealityKit and ARKit provide scene tracking, object detection and tracking, and image tracking. RealityKit is a higher-level framework that builds on top of ARKit. ARKit is a lower-level framework, more flexible than RealityKit, but it requires more code to use.

The SwiftUI view FireCarApp calls ImmersiveView which starts the RealityKit and it loads the Immersive scene into the RealityView which is a SwiftUI view.

RealityView is a SwiftUI view that allows you to display RealityKit content in a SwiftUI app.

In the packages (on the left) you can see the RealityKitContent with the Immersive scene (above the Materials folder). If we click on the Immersive scene we can load the scene in Reality Composer Pro.

Reality Composer Pro

Reality Composer Pro: A Tool for Building 3D and AR Experiences

What it is: A developer tool for designing, editing, and previewing 3D content for use in augmented reality (AR) applications.

What it does:

• Imports and organizes 3D models, materials, and sounds.
• Creates RealityKitContentBundle which combines all assets for Xcode projects.
• Integrates with Xcode for a streamlined workflow.

Key Concepts:

Entity Component System (ECS): A foundational structure for managing 3D content:

• Entities: Basic building blocks representing objects in the scene.
• Components: Define attributes and behaviors of entities (e.g., position, physics).
• Systems: Control interactions and updates between entities and their components.

Custom Components: Allow extending entity functionality beyond pre-defined ones.

Attachments API: Enables embedding interactive SwiftUI views within AR scenes for UI elements.

Dynamic Content Generation: Creating and modifying AR scene elements programmatically based on user interactions or other factors.

Overall, Reality Composer Pro empowers developers to create interactive and adaptable 3D and AR experiences.

The below article is a guide to Reality Composer Pro, a tool for building AR experiences. It covers loading 3D content, the Entity Component System (ECS) for structuring your AR scenes, creating custom components, integrating UI elements using the Attachments API, and generating dynamic content. It also showcases Reality Composer Pro with the Apple Diorama App.

• Reality Composer Pro is a tool for creating 3D and augmented reality (AR) experiences.
• It allows you to import and organize 3D models, materials, and sounds.
• Reality Composer Pro creates a RealityKitContentBundle which includes everything needed to use the 3D content in your Xcode project.
• RealityKit uses the Entity Component System (ECS) for managing 3D content.
• Entities are the basic building blocks, components define attributes and behaviors, and systems control interactions and updates.

import Foundation

// Component: Represents a specific aspect of an entity
struct PositionComponent {
    var x: Float
    var y: Float
}

struct HealthComponent {
    var healthPoints: Int
}

// Entity: Represents an object in the game world
struct Entity {
    var id: Int
    var position: PositionComponent
    var health: HealthComponent
}

// System: Performs operations on entities with specific components
struct MovementSystem {
    func move(entity: inout Entity, deltaX: Float, deltaY: Float) {
        entity.position.x += deltaX
        entity.position.y += deltaY
        print("Entity \(entity.id) moved to (\(entity.position.x), \(entity.position.y))")
    }
}

struct HealthSystem {
    func damage(entity: inout Entity, amount: Int) {
        entity.health.healthPoints -= amount
        print("Entity \(entity.id) took \(amount) damage. Health: \(entity.health.healthPoints)")
    }
}

// Creating entities and systems
var player = Entity(id: 1, position: PositionComponent(x: 0, y: 0), health: HealthComponent(healthPoints: 100))
var enemy = Entity(id: 2, position: PositionComponent(x: 10, y: 5), health: HealthComponent(healthPoints: 50))

let movementSystem = MovementSystem()
let healthSystem = HealthSystem()

// Performing actions using systems
movementSystem.move(entity: &player, deltaX: 2, deltaY: 3)
movementSystem.move(entity: &enemy, deltaX: -1, deltaY: 0)

healthSystem.damage(entity: &player, amount: 20)
healthSystem.damage(entity: &enemy, amount: 10)

• Reality Composer Pro allows you to create custom components to extend the functionality of entities beyond the predefined components provided by the framework.
• The Attachments API allows you to embed SwiftUI views into your 3D AR scenes. This enables you to create interactive UI elements within your AR experience.
• Dynamic content generation refers to the process of creating and modifying elements within an AR scene programmatically. This concept is essential for creating AR experiences that adapt and respond based on user interactions or other factors.

Reality Composer Pro

Omniverse USD Composer vs. Reality Composer Pro

Omniverse USD Composer offers a powerful and highly customizable toolchain, ideal for professional 3D artists working with complex scenes and requiring interoperability across platforms. Reality Composer Pro prioritizes ease of use and real-time performance for AR development within the Apple ecosystem. Its streamlined toolchain might not be suitable for highly detailed 3D assets but excels in rapid prototyping and iteration for AR applications. Both use USD(z) as their foundation to build on.

zoeware - Training

MaterialX

On the bottom of the Reality Composer Pro you will see the MaterialX Shader Graph.

• Aims to provide a common way to represent material definitions across different software and rendering engines.
• Benefits include portability, collaboration, and version tracking of materials.
• Uses node graphs for flexibility and customization.
• Components include shaders, patterns, and surfaces.

Reality Composer Pro

• Uses MaterialX as the foundation for creating custom materials.
• Offers ShaderGraph editor, a visual tool for building materials using nodes.
• Supports two main shader types: Physically Based (PBR) and Custom.
• PBR simulates real-world material properties for realistic appearances.
• Custom shaders offer precise control over object appearance, including animations and effects.

Building Custom Materials

• Done in the ShaderGraph editor by connecting nodes.
• Nodes can represent color manipulation, texture sampling, and more.
• Users can adjust node properties and preview the material in real-time.
• Materials can be reused across different objects in a project.

Additional Concepts

Node Graphs: Visual representations of workflows or systems using connected nodes.

Custom Nodes: User-defined nodes for encapsulating complex functionality.

Geometry Modifiers: Components that dynamically alter the geometry of 3D objects.

Explore MaterialX

Please make sure to grab all the resources (models, materials, ect … fireworks, clouds, sounds, maps) from the Apple sample code to use in your projects.

visionOS | Apple Developer Documentation

What each source example focuses on …

zoeware - Apple Sample Code

USDZ

Here’s a quick overview of OpenUSD:

OpenUSD (USD) is an open-source framework for describing, composing, simulating, and collaborating on 3D worlds.

Originally developed by Pixar, it’s now an industry standard for 3D interchange between different software programs used in visual effects, animation, and other fields like architecture and robotics.

Think of it as a common language for 3D data, allowing seamless exchange of models, textures, materials, and animations across different tools.

Key benefits:

• Collaboration: Enables teams using various software to work on the same 3D scene efficiently.
• Interoperability: Avoids the hassle of data conversion between different 3D applications.
• Version control: Tracks changes and versions of 3D assets for easier management.

OpenUSD uses a scene description approach, where data is organized in a hierarchical structure with elements like geometry, shading, lighting, and physics represented separately.

It leverages node graphs for flexibility and customization, allowing users to build complex scenes by connecting different nodes.

Tools like Pixar’s USD Preview and NVIDIA Omniverse USD are available for working with OpenUSD files.

Overall, OpenUSD provides a powerful and flexible foundation for creating, sharing, and modifying 3D content across various software applications.

Notes on OpenUSD vs USDZ:

zoeware - Training

ARKit

Developed by Apple to create AR experiences on iPhones and iPads. Overlays digital information onto the real world in real-time.

Key features:

• Motion Tracking: Tracks device movement for placing virtual objects.
• Environmental Understanding: Detects surfaces like floors and tables.
• Lighting Estimation: Estimates lighting for realistic shadows and reflections.
• Face Tracking: Tracks facial expressions for adding effects to faces.
• Image and Object Recognition: Triggers AR experiences based on visual markers.
• Integration with SceneKit and Metal: Provides high-quality 3D rendering.
• Multiuser AR: Allows multiple users to interact with virtual objects.
• Persistent AR: Saves and reloads AR experiences.
• LiDAR Support: Enhances AR experiences with more accurate depth sensing.

Review RealityKit

A framework by Apple to simplify AR development on Apple devices.

Review Key features:

• Easy AR Development: Creates AR applications without deep expertise in graphics or 3D programming.
• Entity-Component System: Manages virtual objects with easy customization.
• Realistic Rendering: Applies advanced lighting and shading for realistic visuals.
• Spatial Audio: Creates immersive experiences with sounds from specific virtual positions.
• Swift Integration: Works seamlessly with Swift for clean and expressive code.
• Animations and Physics: Enables dynamic and interactive virtual objects.
• AR Interaction: Allows user interaction with virtual content through gestures and taps.
• SwiftUI Integration: Combines AR experiences with native app UIs.
• AR Quick Look: Lets users preview 3D models directly in AR.
• Xcode Integration: Provides a comprehensive toolkit for building AR applications.

RealityKit & ARKit

Work together to create AR experiences.

• ARKit: Tracks real-world features and provides spatial context.
• RealityKit: Renders and interacts with virtual objects in the AR environment.

Collaboration between ARKit and RealityKit:

• AR Scene Management: ARKit tracks the device and detects features. RealityKit uses this information to place virtual entities.
• Entity-Component System: RealityKit creates and manages virtual entities based on ARKit data.
• Visual Rendering: RealityKit renders virtual entities using lighting and shading information from ARKit.
• Interaction and Animation: ARKit provides user interaction data. RealityKit uses this data to trigger animations and interactions with virtual entities.
• Spatial Audio: Both ARKit and RealityKit support spatial audio.

RealityView in visionOS

• A component in visionOS to integrate RealityKit content into your app.
• Uses RealityViewContentProtocol to manipulate 3D entities.
• RealityView is asynchronous to avoid app hangs while loading content.
• RealityView displays 3D content in true 3D space within your app.

Putting it all together

In visionOS, RealityKit, ARKit, and RealityView work together for AR development:

• RealityKit builds 3D content.
• ARKit understands the real-world context.
• RealityView enables 3D interactions with SwiftUI.

Here’s how these components interact:

1. RealityKit is used to design, customize, and interact with 3D content.
2. ARKit understands the real world environment.
3. RealityView embeds RealityKit content into the SwiftUI interface.
4. You can define interactions and behaviors for 3D content using RealityKit.
5. ARKit provides data to position and anchor RealityKit content in the real world.

RealityView acts as a bridge between RealityKit and ARKit. It allows you to create AR experiences where virtual content interacts with the real world.

Hand Tracking

Only ARKit does hand tracking and it is a bunch of messy code. Use this Apple sample app to look at how hand tracking is done.

Happy Beam | Apple Developer Documentation

When you see crazy code like this …

// Get the position of all joints in world coordinates.
let originFromLeftHandThumbKnuckleTransform = matrix_multiply(
    leftHandAnchor.originFromAnchorTransform, leftHandThumbKnuckle.anchorFromJointTransform
).columns.3.xyz
let originFromLeftHandThumbTipTransform = matrix_multiply(
    leftHandAnchor.originFromAnchorTransform, leftHandThumbTipPosition.anchorFromJointTransform
).columns.3.xyz
let originFromLeftHandIndexFingerTipTransform = matrix_multiply(
    leftHandAnchor.originFromAnchorTransform, leftHandIndexFingerTip.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandThumbKnuckleTransform = matrix_multiply(
    rightHandAnchor.originFromAnchorTransform, rightHandThumbKnuckle.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandThumbTipTransform = matrix_multiply(
    rightHandAnchor.originFromAnchorTransform, rightHandThumbTipPosition.anchorFromJointTransform
).columns.3.xyz
let originFromRightHandIndexFingerTipTransform = matrix_multiply(
    rightHandAnchor.originFromAnchorTransform, rightHandIndexFingerTip.anchorFromJointTransform
).columns.3.xyz


let indexFingersDistance = distance(originFromLeftHandIndexFingerTipTransform, originFromRightHandIndexFingerTipTransform)
let thumbsDistance = distance(originFromLeftHandThumbTipTransform, originFromRightHandThumbTipTransform)


// Heart gesture detection is true when the distance between the index finger tips centers
// and the distance between the thumb tip centers is each less than four centimeters.
let isHeartShapeGesture = indexFingersDistance < 0.04 && thumbsDistance < 0.04
if !isHeartShapeGesture {
    return nil
}


// Compute a position in the middle of the heart gesture.
let halfway = (originFromRightHandIndexFingerTipTransform - originFromLeftHandThumbTipTransform) / 2
let heartMidpoint = originFromRightHandIndexFingerTipTransform - halfway


// Compute the vector from left thumb knuckle to right thumb knuckle and normalize (X axis).
let xAxis = normalize(originFromRightHandThumbKnuckleTransform - originFromLeftHandThumbKnuckleTransform)


// Compute the vector from right thumb tip to right index finger tip and normalize (Y axis).
let yAxis = normalize(originFromRightHandIndexFingerTipTransform - originFromRightHandThumbTipTransform)


let zAxis = normalize(cross(xAxis, yAxis))


// Create the final transform for the heart gesture from the three axes and midpoint vector.
let heartMidpointWorldTransform = simd_matrix(
    SIMD4(xAxis.x, xAxis.y, xAxis.z, 0),
    SIMD4(yAxis.x, yAxis.y, yAxis.z, 0),
    SIMD4(zAxis.x, zAxis.y, zAxis.z, 0),
    SIMD4(heartMidpoint.x, heartMidpoint.y, heartMidpoint.z, 1)
)
return heartMidpointWorldTransform

Your looking at hand tracking code … and it will not run in the simulator.

Apple will make this easier over time. For now replace the hand tracking with buttons … move right/left/up/down/click buttons.

Tools

• Reality Composer Pro: Creates and previews 3D content for visionOS apps.
• Reality Composer for iOS/iPadOS: Builds, tests, and simulates AR experiences.
• Reality Converter: Converts, views, and customizes USDZ 3D objects on Mac.

Introducing Reality Converter - Latest News - Apple Developer

• Object Capture

Make models with your iPhone/iPad

Object Capture - Augmented Reality - Apple Developer

• RoomPlan

Introducing RoomPlan — Powered by ARKit, RoomPlan is a new Swift API that utilizes the camera and LiDAR Scanner on iPhone and iPad to create a 3D floor plan of a room, including key characteristics such as dimensions and types of furniture.

RoomPlan - Augmented Reality - Apple Developer

Thanks for coming to the talk and good luck at the hackathon!

List of articles:

Siamak (Ash) Ashrafi

Apple Vision Pro - visionOS Development

View list

9 stories

This article covers everthing about visionOS programming but is almost impossible to read … we will be breaking it up soon …

Developing for Apple VisionOS

~Ash

Building the Future!