Every capability,
documented.
PilotGentic ships as six complete capability streams — from the SQLite foundation layer all the way to remote phone control. Streams 0–3 are free. Streams 4–5 are paid add-ons for Pro and Max subscribers.
Stream 0
Foundation
The bedrock everything else runs on.
- SQLite behavior database — FTS5 full-text search over all recorded actions
- AX compare — accessibility tree diffing to detect UI changes between frames
- Action recorder — step-by-step workflow capture with typed event payloads
- Migration system — zero-downtime schema evolution as the database grows
Stream 1
Task Engine
Persist, version, and run automation tasks at scale.
- Task store — persist, version, and manage automation tasks in SQLite
- Task runner — execute tasks step-by-step with live per-step status updates
- Task escalation — smart interrupt handling that pauses and prompts Claude when input is required
- Task tools — pilotgentic_task MCP suite: create, run, list, delete from any Claude session
Stream 2
Video Upload Path
Show Claude a screen recording — get a working task.
- Frame extractor — pulls keyframes from uploaded screen recordings at optimal density
- Analyzer — Claude Vision interprets each frame's UI state and interactive elements
- Synthesizer — converts frame-by-frame analysis into ordered, executable task steps
- AX verifier — validates every synthesized step against the live accessibility tree before saving
Stream 3
Swift UI
First-class task management built into the app.
- Task Dashboard — full SwiftUI task management panel embedded directly in the app sidebar
- Task Wizard — guided step-by-step interface for training new tasks without writing code
- Progression Meter — gamified L1–L4 AI skill level displayed live in the sidebar
- MCPClient — direct Swift-to-MCP bridge enabling in-app tool calls without round-tripping Claude
Stream 4
ML Layer
Anomaly detection, semantic search, and optimization proposals.
- Anomaly Detector — Z-score anomaly detection on task run history flags regressions automatically
- Task Optimizer — 3-pass analysis (wait reduction, tab order, dialog skip) proposes improvements, never auto-applies
- Progression Engine — L1–L4 leveling system with prerequisite tracking and unlock gates
- Voyage Indexer — semantic vector search over all action events via a 5-minute background job (Voyage AI)
Stream 5
Remote Add-On
Control your Mac from anywhere. Get notified the moment Claude needs you.
- Relay server — WebSocket relay on Fly.io (pilotgentic-registry.fly.dev) bridges your Mac and phone
- QR pairing — scan a QR code on your phone to pair with your Mac in under 2 minutes, HMAC-signed tokens
- Push notifications — Web Push (VAPID) alerts delivered the moment Claude needs your input
- Task dispatch — trigger any trained task from your phone with live per-step status updates
- Supervisor View — manage multiple Macs from one dashboard with a global escalation queue
- Screen streaming — live JPEG frame stream from Mac to phone at configurable quality (high / medium / low)
- Session revoke — OTP-authenticated session revocation with email confirmation
Add-On
Voice
Hands-free conversation with Claude — on-device STT and AI voice output.
- Whisper STT — on-device speech-to-text via whisper-cli — no cloud, no API key required for voice input
- ElevenLabs AI voice — natural AI voice output via ElevenLabs streaming API, or on-device AVSpeechSynthesizer fallback
- Hands-free mode — mic auto-re-arms after Claude finishes speaking for a continuous conversation loop
- Voice ring UI — 48-bar animated ring — cyan while listening, electric blue while speaking
ML Layer
Anomaly detection, semantic search, and optimization proposals.
- Anomaly Detector
- Task Optimizer
- Progression Engine
- Voyage Indexer
Remote Add-On
Control your Mac from anywhere. Get notified the moment Claude needs you.
- Relay server
- QR pairing
- Push notifications
- Task dispatch
- Supervisor View
- Screen streaming
- Session revoke
Voice
Hands-free conversation with Claude — on-device STT and AI voice output.
- Whisper STT
- ElevenLabs AI voice
- Hands-free mode
- Voice ring UI
Add-ons are modular and downloadable — the base app stays lean. Compare plans →
What you get
Free, and more when you need it
Start with everything free.
Streams 0–3 ship with the free download. Unlock the ML and Remote add-ons when you are ready to go further.
macOS 14 Sonoma or later · Apple Silicon & Intel