13 streams · All capabilities documented
Every capability,
documented.
PilotGentic ships as 13 capability streams — from the SQLite foundation layer to remote phone control, in-browser widgets, and the precision action engine. Most streams are free; ML, Remote, and Voice are paid add-ons for Pro and Max subscribers. Anything not yet working end to end is labelled “In development” — no exceptions.
Stream 0
Foundation
The bedrock everything else runs on.
Free tier
- SQLite behavior database — every action PilotGentic takes is recorded and searchable — Claude can look up what happened, when, and why — full-text search via FTS5
- AX compare — detects when a UI changes between steps — catches broken automations before they fail silently — accessibility tree diffing at the frame level
- Action recorder — captures exactly what you do step-by-step so Claude can learn from it — typed event payloads, not fragile screenshots
- Migration system — the database evolves as PilotGentic grows — schema updates happen automatically, no data loss, no manual steps
Stream 1
Task Engine
Persist, version, and run automation tasks at scale.
Free tier
- Task store — your automations are saved and versioned — run them again anytime, track every change, nothing gets lost between sessions
- Task runner — watch your task execute live — each step updates in real time so you always know where it is and what it's doing
- Task escalation — Claude pauses and asks you when it hits a decision it can't make — you unblock it, it continues — no babysitting required
- Task tools — create, run, list, and delete tasks directly from Claude — just ask, no separate UI needed — pilotgentic_task MCP suite
Stream 2
Video Upload Path
Show Claude a screen recording — get a task it can learn from.
In developmentFree tier
In development — under active testing. Frame extraction, Vision analysis, and step synthesis run today; the final step, executing a video-trained task end to end on your live screen, is still being hardened and has not shipped yet.
- Frame extractor — record yourself doing a task once — PilotGentic pulls the key moments automatically, no manual annotation needed
- Frame filmstrip + per-frame prompts — review every extracted frame in a filmstrip and attach your own note to each one — steer the analysis before synthesis runs
- Analyzer — Claude Vision reads every screen state in your recording — understands what each UI element is and what you did with it
- Synthesizer In development — turns the analysis into a step-by-step automation draft — no code written
- AX verifier — checks every generated step against your live UI before saving — catches anything that won't work on your actual screen
- End-to-end execution In development — running a video-trained task on your live screen — the synthesis-to-execution contract is under active testing and has not completed a real task yet
Stream 3
Swift UI
First-class task management built into the app.
Free tier
- Task Dashboard — see all your automations at a glance — run, edit, or delete from the sidebar without opening Claude
- Task Wizard In development — train a new task by walking through it step by step — no code, no prompt engineering, just show PilotGentic what to do — wizard-trained runs are under active testing
- Progression Meter — tracks how capable your AI setup is getting — L1 to L4 skill level shown live, unlocks as your automation library grows
- MCPClient — the app can call its own tools directly — no round-trip to Claude needed for in-app actions, keeps the UI fast and responsive
Stream 4
ML Layer
Anomaly detection, semantic search, and optimization proposals.
Pro / Max add-on
- Anomaly Detector — notices when a task starts taking longer or failing more than usual — flags it automatically so you can fix it before it breaks — Z-score analysis on run history
- Task Optimizer — analyzes your automations and suggests concrete speed improvements — shorter waits, better click order, skipped dialogs — proposes, never auto-applies
- Progression Engine — tracks your automation library as it grows — prerequisite-aware L1–L4 leveling unlocks more advanced capabilities as your skills develop
- Voyage Indexer — find any past action by describing it in plain English — semantic search over everything PilotGentic has ever done, powered by Voyage AI
Stream 5
Remote Add-On
Control your Mac from anywhere. Get notified the moment Claude needs you.
Pro / Max add-on
- Relay server — works from anywhere — no port forwarding, no VPN — WebSocket relay on Fly.io bridges your Mac and browser through NAT
- QR pairing — pair a new device in under 30 seconds — scan the QR code in Settings, done — HMAC-signed tokens, 2-min expiry
- Push notifications — get alerted the moment Claude needs your input — Web Push (VAPID) delivered to your phone even when the tab is closed
- Task dispatch — trigger any trained automation from your phone and watch it run step-by-step — no need to be at your Mac
- Supervisor View — manage multiple Macs from one browser dashboard — global escalation queue shows every Mac waiting for input
- Screen streaming — see exactly what's on your Mac screen in real time — JPEG stream at configurable quality, toggle on demand to save bandwidth
- Remote Terminal — open a shell on your Mac from any browser, anywhere — run commands, check logs, start builds — full PTY with color, resize, and no SSH setup needed
- Input forwarding — click, type, and scroll directly in the screen stream — your Mac responds as if you're sitting at it — CGEvent dispatch, not screen sharing lag
- Session revoke — revoke any paired device immediately — OTP-authenticated with email confirmation so only you can do it
Stream 7
Session Mode
Choose how AI acts on your Mac — observable or silent, your call.
Free tier
- AI Doppler mode — cursor moves visibly to every target — you see exactly what Claude is doing, step by step, in real time
- Background mode — silent AX press with no cursor movement — Claude acts without interrupting your screen, ideal for unattended or parallel workflows
- Hybrid mode — observable for the interactions you care about, silent for the rest — mix visibility and throughput in the same session
- Strict enforcement — hard gate: if a tool call would violate the declared mode, it is blocked and returned as an error — no silent fallbacks
- Flexible enforcement — soft gate: mode violations are logged and allowed — useful while developing or when you want awareness without blocking
- Per-session policy — mode and enforcement level are declared in CLAUDE.md and injected into every session — Claude Code picks them up automatically
Add-On
Voice
Hands-free conversation with Claude — on-device STT and AI voice output.
Pro / Max add-on
- Whisper STT — on-device speech-to-text via whisper-cli — no cloud, no API key required for voice input
- ElevenLabs AI voice — natural AI voice output via ElevenLabs streaming API, or on-device AVSpeechSynthesizer fallback
- Hands-free mode — mic auto-re-arms after Claude finishes speaking for a continuous conversation loop
- Voice ring UI — 48-bar animated ring — cyan while listening, electric blue while speaking
Stream 8
Dev Loop
Edit. Reload. Test. 3 seconds, same session — no build cycle, no context loss.
Free tier
- Hot-reload MCP harness — edit any tool handler in dev source — the MCP server picks it up in ~3 seconds via node --watch, same Claude session continues uninterrupted
- HTTP transport — MCP over HTTP/SSE lets Claude reconnect transparently after a server restart — no session kill, no context loss, no 90-second build cycle for JS changes
- Trajectory-based skill extraction — behavior.db records every action with outcome, timing, and AX labels — external agents can read failure trajectories and propose targeted fixes
- Parallel dev + prod servers — run the dev hot-reload server alongside the stable bundle — test new tool logic on port 7778 while the production bundle stays on stdio
- Three-surface sync checker — the tool registry, the skills catalog, and the agent whitelist must agree — an automated checker catches drift between them before it ships
Stream 9
Browser DevTools
Claude sees your browser the way DevTools does — console, network, and the live DOM.
Free tier
- Live console capture — read errors, warnings, and logs from any open tab — Claude sees the exact stack trace, not a screenshot of it
- Network inspection — capture requests and responses as they happen — status codes, payloads, timing — debug an API from the chat
- JS execution in the page — run JavaScript in the page's own world — query state, click React components, read variables no screenshot can see
- Extension self-reload — the Chrome extension updates itself in place — ship a fix, reload, keep the same tab and session
- Native messaging bridge — Desktop and PG Page buttons in the extension talk directly to the Mac app — one click jumps from a web page to the desktop app and back
Stream 10
Widget Suite
A live PilotGentic control surface inside any browser tab.
Free tier
- The PG bar — an open-carry strip pinned to the top of the page — four action buttons plus a master toggle, always in reach, and it survives page reloads
- Control Center — the command hub widget — open the other widgets, flip the master toggle, and keep PilotGentic one click away in any tab
- Stats overlay — live session metrics floating over the page — see what PilotGentic is doing without leaving your work
- Debug Footer — console and network health pinned to the bottom of the page — spot the error the moment it happens
- Chat widget — a real Claude CLI session — not a chat mock-up: a persistent PTY running Claude Code in your browser tab — reload the page, the session is still there
- One design system — every widget shares the brand header — green connected dot plus the PilotGentic wordmark — and grab-anywhere resize handles
Stream 11
Action Engine
The precision layer under every click — verify before, act, verify after.
Free tier
- The act chain verb — one call runs a whole sequence — locate, click, type, verify — with per-step run tracing so every action is auditable afterward
- Ranked locate candidates — when an element is ambiguous, PilotGentic returns scored candidates instead of guessing — the best match wins, the rest are on record
- Bounds gating + visibility flags — a click only fires if the target is actually on screen, visible, and inside its window bounds — no more clicking where a button used to be
- Browser-aware AX hints — the accessibility layer knows when it's looking at a browser and routes to the DOM instead — native windows and web pages each get the right instrument
- Post-settle snapshots — after every action the UI is re-captured once it settles — the next step starts from what the screen actually shows, not what it showed before
- Terminal-condition signals — actions declare what 'done' looks like — success and failure are detected, not assumed
Platform
Accounts & Licensing
Sign in once — your license and tier follow your account, not your browser.
Free tier
- Desktop OAuth — sign in with Google or GitHub directly inside the Mac app — no copying keys out of an email
- Automatic license import — sign in and your license and tier are pulled in automatically — Pro and Team unlock without a manual activation step
- Account-recoverable licenses — the key follows your account, not a browser or a machine — new Mac, cleared cache, reinstall: sign in and it's back
Stream 4 · Pro / Max
ML Layer
Anomaly detection, semantic search, and optimization proposals.
- Anomaly Detector
- Task Optimizer
- Progression Engine
- Voyage Indexer
Stream 5 · Pro / Max
Remote Add-On
Control your Mac from anywhere. Get notified the moment Claude needs you.
- Relay server
- QR pairing
- Push notifications
- Task dispatch
- Supervisor View
- Screen streaming
- Remote Terminal
- Input forwarding
- Session revoke
Stream 6 · Pro / Max
Voice
Hands-free conversation with Claude — on-device STT and AI voice output.
- Whisper STT
- ElevenLabs AI voice
- Hands-free mode
- Voice ring UI
Add-ons are modular and downloadable — the base app stays lean. Compare plans →
What you get
Free, and more when you need it
StreamFreePro / Max
Foundation
Task Engine
Video Upload Path
Swift UI
ML Layer
—
Remote Add-On
—
Session Mode
Voice
—
Dev Loop
Browser DevTools
Widget Suite
Action Engine
Accounts & Licensing
Start with everything free.
Streams 0–3 ship with the free download. Unlock the ML and Remote add-ons when you are ready to go further.
macOS 14 Sonoma or later · Apple Silicon & Intel