Foundations of AI/ML on Mobile
Creating Magic with ML on Mobile
It’s amazing what we can build with ML on mobile these days. Here is a quick review for you consideration.
AI/ML on Smartphone
The modern smartphone has crazy hardware specs, software frameworks add tools we can use in our ML apps.
Hardware
Let’s review keeping in mind what AI/ML needs as input.
Apple(iOS)
- Sensors: GPS, Accelerometer, Gyroscope, magnetometer, barometer, proximity sensor, Fingerprint Sensor, ambient light sensor, 5G, WiFi 6, Bluetooth 5.2 and NFC.
- Camera: Ultra Wide, Telephoto, Lidar Scanner
- Processor and Chipset: CPU: A16 Bionic chip, Neural Engine: 16-core
Google(Android)
- Sensors: GPS, Accelerometer, Gyroscope, Magnetometer, Barometer, Proximity Sensor, Ambient Light Sensor, Fingerprint Sensor (may vary by model), 5G, WiFi 6, Bluetooth 5.2 and NFC.
- Camera:Ultra Wide and Telephoto lenses.
- Processor and Chipset: CPU: Varies by manufacturer (e.g. Snapdragon 8 Gen 1), Neural Processing Unit (NPU)
Software
The modern smartphone has advanced software frameworks we can use in our ML apps. Let’s review keeping in mind what AI/ML needs as platform.
Apple (iOS / watchOS / visionOS)
- Core Frameworks: Swift · SwiftUI · SF Symbols · SwiftData · Swift Structured Concurrency · Combine · Swift Package Manager · XCTest · CI/CD (GitHub Actions)
- App Frameworks :Swift Charts · CoreML · Maps · ARKit · Apple Health
- visionOS Frameworks: RealityKit · RealityView · ARKit · Reality Composer Pro · MaterialX · USDZ
Android (wearOS)
- Core Frameworks: Kotlin · Kotlin Coroutines (Flow/Channels) · Jetpack Compose (Material 3) · Hilt (ViewModel, Navigation, Room, Testing) · Room · Coil · Workmanager · Junit4 · CI/CD (GitHub Actions)
- App Frameworks: Maps · Health Connect · CameraX · BLE · ML Kit · ARCore
Now we see the inventory of tech lets see what we can build with it.
ML projects on Smartphones
Let review some mind blowing 🤯 projects on SmartPhone using the above hardware/software.
- Hummingbird: This AI-powered mobile app utilizes machine learning to analyze health data and recommend personalized microdosing schedules for medication.
- Photodo: This AI-powered to-do app lives conveniently on your phone. Simply capture a photo, record a voice memo, or take a note, and Photodo’s intelligent assistant swings into action. It identifies the task, sets deadlines, locates nearby assistance, and even tracks your budget. It’s like having a personal AI assistant in your pocket. (Use Gemini Nano AI on mobile)
- RxDigita: Streamlining medication management, RxDigita allows you to add meds by voice or photo (the app reads the label!), set reminders, and keep track of everything in a clear and organized list view.
- AshBike: This eco-friendly e-bike service leverages your fitness data (via Apple Health and Google Fit integration). Its ML capabilities help plan routes based on your calendar schedule and reward you with NFTs for choosing sustainable transportation
- Building a self-driving car with nothing but a Pixel Phone!
- Building a robot/drone with an Android Phone
- Rust for ROTS
This would make an awesome new product for Lego.
Articles — AI/ML on Mobile
ML on smartwatch.
Lets talk about why the smartwatch will change how humanity lives.
Discover the captivating journey of smartwatches as they transcend from mere gadgets to indispensable lifestyle companions. With global shipments set to skyrocket, these sleek and stylish devices are revolutionizing communication, health management, and fitness.
- SmartWatch Development (with ML)
ML projects on Smartwatch
- DrunkWatch: This innovative system leverages your smartwatch to assess your intoxication level and determine your driving suitability. It analyzes your walking patterns, writing style (when entering a code), and bio-sensor data to make this crucial call. This showcases the combined power of the iWatch’s 4-core Neural Engine, accelerometer, and bio-sensors.
watchOS / iWatch10:
Android Hand Recognition using TF Lite
- Hummingbird: Uses the iWatch / WearOS sensors for real-time ML physiological data to make the determination of when to take the medications. (See above 👆)
- AshBike: Uses the smartwatch to track the person and the workout.
- ZoeWare: Uses the smartwatch in ML biofeedback for mediation.
Getting The Models
The hardest part of any ML project is getting good data … here are some resources to help
TensorFlow Hub
Models & datasets — Explore repositories and other resources to find available models, modules and datasets created by the TensorFlow community.
Kaggle
Datasets — Explore, analyze, and share quality data. Learn more about data types, creating, and collaborating.
OpenML
Datasets provide training data for machine learning models. OpenML datasets are uniformly formatted and come with rich meta-data to allow automated processing. You can sort or filter them by a range of different properties. Learn more.
Hugging Face
The platform where the machine learning community collaborates on models, datasets, and applications.
ModelZoo
Model Zoo curates and provides a platform for deep learning researchers to easily find pre-trained models for a variety of platforms and uses. We regularly update the site, and provide filtering functionality for users to find models that they need, either for educational purposes, transfer learning, or other uses.
Papers with Code
The mission of Papers with Code is to create a free and open resource with Machine Learning papers, code, datasets, methods and evaluation tables.
Open Model Zoo
This repository includes optimized deep learning models and a set of demos to expedite development of high-performance deep learning inference applications. Use these free pre-trained models instead of training your own models to speed-up the development and production deployment process.
V7Labs Full list
Save time searching for quality training data for your machine learning projects, and explore our collection of the best free datasets.
Model Search
Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers speed up their exploration process for finding the right model architecture for their classification problems (i.e., DNNs with different types of layers).
For iOS
Awesome CoreML Models
Coding AI/ML on Android & iOS
Compare and contrast all the various tools ML tools we can use to build our apps.
Android
- ML Kit: Perfect for quick integration. It offers user-friendly APIs and pre-built models for common tasks like face detection or text recognition. However, customization is limited.
- Firebase ML (Deprecated — See Cloud Vision APIs)
- TensorFlow (TF) Lite: If high performance and complete control are your priorities, TF Lite is the way to go. It allows you to leverage custom models, but requires a solid understanding of machine learning concepts.
- MediaPipe (New): This newcomer simplifies complex machine learning pipelines by offering a versatile and easy-to-use framework built on top of TF Lite. For developers seeking a low-code or no-code alternative to TF Lite, MediaPipe is a great option.
- Gemini Nano (New): Privacy takes center stage with Gemini Nano. This secure execution environment boasts a chat-like interface for interacting with models. However, current limitations include flexibility and device compatibility (currently only available on Pixel 8 Pro).
- AI Core (New): This hardware acceleration component complements Gemini Nano on compatible devices, further optimizing its performance.
Very detailed article on coding / building ML on Android …
iOS
Core ML and Create ML — A Powerful Duo
Apple’s mobile ecosystem offers Core ML and Create ML for streamlined ML development on iOS.
- Core ML: The core (pun intended) of on-device machine learning in iOS apps. It acts as a bridge, allowing your app to interact with various machine learning models in a unified way. These models can be used for tasks like image classification or making predictions. Core ML empowers you to run models directly on the device, enhancing speed and privacy compared to cloud-based solutions.
- Create ML: This user-friendly tool streamlines the creation of custom machine learning models. Imagine it as a drag-and-drop interface where you provide your training data, and Create ML handles the model creation process. It’s ideal for beginners or scenarios where complex models aren’t required.
In the next section we will review the code for building all this … stay tuned!
~Ash
P.S. This will be talk presented @ GDG San Jose event: Dual Event — Build With AI and IWD 2024
Title
Don’t Be a Phone-y: Building Smart Mobile Apps with AI/ML
Summary
Imagine having a powerful AI assistant living right in your pocket. Your phone can already take amazing pictures and track your workouts, but what if it could analyze your health data and recommend personalized medication schedules, or understand your tasks and deadlines like a super-powered to-do list? This is the exciting world of AI-powered mobile apps, and it’s not science fiction anymore. In this talk, we’ll unlock the secrets to building these intelligent applications. We’ll explore how modern smartphones are packed with the processing power and sensors to run AI on-device, and dive into the different tools and frameworks available to turn your ideas into reality. Get ready to unleash the true potential of your phone and discover how AI is revolutionizing mobile development. This talk is equal parts theory and code.
