Skip to content
Back to search

Jarvis-Alsis

Jarvis Desktop Assistant Event-driven macOS Desktop AI Assistant powered by Gemini, Official MCP Servers, Vision GUI Automation, and Dual-Engine STT (Groq + Faster-Whisper).

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/tanachai015--jarvis-alsis/badge.svg)](/listings/tanachai015--jarvis-alsis)

HTML

<a href="/listings/tanachai015--jarvis-alsis"><img src="/api/listings/tanachai015--jarvis-alsis/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Clone & Setup Environment

git clone <your-repo-url>
cd "Javis Project"

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

2. Configure Environment (`.env`)

cp .env.example .env

Edit `.env` with your API keys:

# Google Gemini API Key (Required for Brain & Vision)
GEMINI_API_KEY=AIzaSy...

# Groq Cloud API Key (Recommended for ultra-fast ~0.2s STT)
GROQ_API_KEY=gsk_...
STT_MODE=groq
GROQ_WHISPER_MODEL=whisper-large-v3-turbo

# Voice Listening Mode: hybrid, push_to_talk, or hi_jarvis
VOICE_LISTEN_MODE=hybrid

3. Launch Jarvis

python3 main.py

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 231 lines
  1. # Jarvis Desktop Assistant 🚀
  2. [![Platform](https://img.shields.io/badge/Platform-macOS%2014%2B%20(Tahoe%20%2F%20Sonoma%20%2F%20Sequoia)-blue?logo=apple)](https://www.apple.com/macos/)
  3. [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/)
  4. [![GUI](https://img.shields.io/badge/UI-PySide6%20(Qt6)-41CD52?logo=qt)](https://www.qt.io/)
  5. [![AI Engine](https://img.shields.io/badge/AI-Google%20GenAI%20SDK-4285F4?logo=google)](https://aistudio.google.com/)
  6. [![Protocol](https://img.shields.io/badge/Standard-Model%20Context%20Protocol%20(MCP)-8B5CF6)](https://modelcontextprotocol.io/)
  7. [![Speech](https://img.shields.io/badge/STT-Groq%20Cloud%20%2B%20Faster--Whisper-F55036)](https://groq.com/)
  8. [![License](https://img.shields.io/badge/License-MIT-green)](LICENSE)
  9. An agentic, event-driven, multimodal desktop AI companion built natively for macOS. Designed with a **decoupled PySide6 architecture**, a **Static Tool Registry combining Native macOS tools with Official Model Context Protocol (MCP) servers**, a **dual-engine hybrid Speech-to-Text pipeline**, **computer vision desktop automation**, and **100% stealth background execution**.
  10. ---
  11. ## 🌟 Key Features & Engineering Highlights
  12. ### 1. 🎙️ Dual-Engine Hybrid STT Pipeline (~0.2s Latency)
  13. - **Primary Cloud Engine:** Integrates **Groq Cloud STT (`whisper-large-v3-turbo`)** via in-memory 16kHz WAV streaming (`io.BytesIO`), achieving sub-second response times (~0.2s - 0.4s) at **0% local CPU usage**.
  14. - **Automatic Local Fallback:** Seamlessly fails over to an on-device **`Faster-Whisper` (`small`, int8)** model if offline, rate-limited, or network-disconnected without interrupting the user.
  15. - **Smart Pacing & Silence Auto-Cutoff:** Real-time RMS audio energy detection automatically cuts off and submits queries after **3.5 seconds of silence** (`VOICE_POST_SPEECH_SILENCE`), with a 15-second maximum safety timeout.
  16. ### 2. 🎛️ 3 Adaptive Voice Listening Modes (Real-Time Switcher)
  17. - **`push_to_talk`**: Triggers recording solely via macOS system hotkey (**`Cmd + Shift + J`**), consuming 0% mic CPU when idle.
  18. - **`hi_jarvis`**: Silent background wake-word detection powered strictly by **Local `Faster-Whisper`** (preserving Groq API quota). Supports *"Hi Jarvis"*, *"Jarvis"*, *"จาร์วิส"*, *"จาวิส"*, and *"ไฮ จาร์วิส"*.
  19. - **`hybrid` (Default)**: Dual trigger — seamlessly activate via hotkey or speak the wake word anytime.
  20. - **Runtime Mode Switcher:** Switch modes instantly without application restart via the macOS Menu Bar Status Item.
  21. - **RAM Lifecycle Management:** Pre-loads model in RAM for `hi_jarvis`/`hybrid`, and immediately unloads with `gc.collect()` when switching to `push_to_talk` or quitting.
  22. ### 3. 👁️ macOS Vision GUI Automation (Retina-Aware Mouse & Keyboard)
  23. - **`click_on_screen`**: Captures screen, prompts Gemini Vision for coordinates, and calculates exact **Retina Display scale factors** (`img_w / screen_w`) for 100% accurate pixel-to-point mouse clicking.
  24. - **Inactive Window Focus Handling**: Automatically performs a two-stage click sequence (focus click + 0.15s window manager delay + action click) guaranteeing background windows (e.g. Firefox, Chrome) activate and play videos/links immediately.
  25. - **`type_text_on_screen`**: Injects text via macOS Clipboard (`pbcopy` + `Cmd+V`) supporting **Thai, English, Unicode, and Emoji** without dropping keystrokes, with asynchronous clipboard restoration.
  26. - **`click_and_type`**: Single-step GUI locator, window focus activator, and keyboard input tool.
  27. ### 4. 🛠️ Static Tool Registry & Official MCP Ecosystem (40 Tools)
  28. - **Pre-Compiled Tool Registry:** Native tools and MCP tools are pre-compiled into `GenerateContentConfig` once at startup, eliminating per-request schema construction overhead.
  29. - **Instant Boot (0.005s):** Persistent tool schema cache (`memory/mcp_tools_cache.json`) with SHA-256 configuration checksum validation.
  30. - **Official MCP Servers:** Runs standard Stdio sub-processes:
  31. - `@modelcontextprotocol/server-filesystem` (16 tools)
  32. - `mcp-server-fetch` (1 tool)
  33. - `mcp-server-git` (14 tools)
  34. - Native macOS Tools: Vision, System Control, Antigravity Bridge, Memory, Reveal, Raw History, Mouse & Keyboard (9 tools)
  35. ### 5. 🥷 100% Stealth Background Execution & On-Demand Reveal
  36. - Never steals desktop focus during background file edits, code refactoring, or IDE task delegations.
  37. - **On-Demand Reveal (`reveal_task_result`):** Results, modified files, and browser outputs open on-screen **only when explicitly commanded by the user** (e.g. *"เปิดดูผลงานล่าสุด"*, *"reveal"*).
  38. ### 6. 🗣️ Dual-Output Protocol & Continuous Bilingual Speech
  39. - **Visual Display (Overlay Chat):** Full comprehensive Markdown, tables, and fenced code blocks.
  40. - **Natural Spoken Speech (Edge-TTS):** Clean spoken summary inside `[SPEAK]` tags across multi-line paragraphs without premature sentence cutoffs.
  41. - **Bilingual Tone:** Automatic script detection toggling between Thai (`th-TH-NiwatNeural`, -14Hz) and British Butler English (`en-GB-RyanNeural`, -8Hz).
  42. ---
  43. ## 🏛️ System Architecture
  44. ```text
  45. [User Trigger: Cmd+Shift+J / "Hi Jarvis"]
  46. │
  47. ▼
  48. [Quartz CGEventTap / sounddevice Mic]
  49. │
  50. ├── Mode: push_to_talk ──> Hotkey only
  51. ├── Mode: hi_jarvis ──> Local Faster-Whisper (Wake Word)
  52. └── Mode: hybrid ──> Both triggers active
  53. │
  54. ▼
  55. [Speech-to-Text Pipeline]
  56. │
  57. ┌───────────┴───────────┐
  58. ▼ ▼
  59. [Primary Cloud STT] [Local Fallback STT]
  60. Groq whisper-large Faster-Whisper
  61. (~0.2s latency) (Offline / int8)
  62. │ │
  63. └───────────┬───────────┘
  64. │ (emit user_message_received)
  65. ▼
  66. [JarvisBrain (core/brain.py)]
  67. │
  68. ┌───────────┴───────────┐
  69. ▼ ▼
  70. [Explicit Fallback Chain] [Static Tool Registry (40 Tools)]
  71. gemini-3.5-flash-lite • Native macOS Tools (9)
  72. gemini-3.1-flash-lite - Vision Screen Analysis
  73. gemini-3.6-flash - Retina Click / Clipboard Type
  74. gemini-3.8-flash - System Control & Memory
  75. - Stealth Reveal & Raw Archive
  76. • Official MCP Servers (31)
  77. - filesystem, fetch, git
  78. │
  79. ▼
  80. [Dual-Output Protocol]
  81. │
  82. ┌───────────┴───────────┐
  83. ▼ ▼
  84. [Visual Screen HUD] [Neural Voice Synthesis]
  85. OverlayChat Window Microsoft Edge-TTS
  86. (Markdown, Code) (Dual-Tone TH/EN afplay)
  87. ```
  88. ---
  89. ## 📂 Project Directory Structure
  90. ```text
  91. Javis Project/
  92. ├── .env # Local environment secrets (Git-ignored)
  93. ├── .env.example # Safe environment configuration template
  94. ├── .gitignore # Strict security & cache ignore rules
  95. ├── requirements.txt # Python dependencies
  96. ├── config.py # Central configuration & runtime constants
  97. ├── main.py # Entry point & application lifecycle manager
  98. ├── mcp_servers.json # Official MCP server configurations
  99. ├── PROJECT_ARCHITECTURE.md # In-depth architectural blueprint
  100. ├── CHANGELOG.md # Release version history
  101. ├── README.md # Portfolio showcase & documentation
  102. │
  103. ├── core/ # Core system logic
  104. │ ├── event_bus.py # Thread-safe PySide6 signal bus
  105. │ ├── brain.py # Gemini cognitive processor & fallback chain
  106. │ ├── tools.py # Native macOS tools (Vision, Mouse, Keyboard)
  107. │ ├── task_manager.py # Stealth task registry & on-demand reveal
  108. │ ├── raw_archive.py # 365-day monthly partitioned JSONL raw storage
  109. │ └── mcp_manager.py # MCP client & persistent schema disk cache
  110. │
  111. ├── integrations/ # External developer integrations
  112. │ └── antigravity_bridge.py # Silent file-based task bridge + Watchdog
  113. │
  114. ├── voice/ # Voice I/O pipeline
  115. │ ├── listener.py # Dual-engine STT, 3 listening modes, silence cutoff
  116. │ └── speaker.py # Bilingual dual-tone Edge-TTS + macOS afplay
  117. │
  118. ├── ui/ # PySide6 desktop interface
  119. │ ├── dynamic_island.py # Floating macOS pill status indicator
  120. │ ├── overlay_chat.py # Spotlight/Raycast style chat HUD
  121. │ └── tray.py # Menu bar status icon & mode switcher
  122. │
  123. └── memory/ # Local data storage (Git-ignored)
  124. └── .gitkeep # Repository directory anchor
  125. ```
  126. ---
  127. ## 🛠️ Prerequisites & macOS Permissions
  128. To enable seamless low-level system interaction on macOS:
  129. 1. **Accessibility (`AXIsProcessTrustedWithOptions`)**:
  130. - Required for Quartz System EventTap (`Cmd+Shift+J`, `Alt+Space`) and synthetic mouse/keyboard automation.
  131. - Go to: **System Settings > Privacy & Security > Accessibility**
  132. - Enable: **Terminal / iTerm / IDE / Python**
  133. 2. **Microphone**:
  134. - Required for `voice/listener.py`.
  135. - Go to: **System Settings > Privacy & Security > Microphone**
  136. - Enable your terminal or IDE.
  137. 3. **Screen Recording**:
  138. - Required for `screencapture -x` vision analysis and GUI coordinate detection.
  139. - Go to: **System Settings > Privacy & Security > Screen Recording**
  140. - Enable your terminal or IDE.
  141. ---
  142. ## ⚡ Quickstart & Installation
  143. ### 1. Clone & Setup Environment
  144. ```bash
  145. git clone <your-repo-url>
  146. cd "Javis Project"
  147. # Create and activate virtual environment
  148. python3 -m venv .venv
  149. source .venv/bin/activate
  150. # Install dependencies
  151. pip install -r requirements.txt
  152. ```
  153. ### 2. Configure Environment (`.env`)
  154. ```bash
  155. cp .env.example .env
  156. ```
  157. Edit `.env` with your API keys:
  158. ```ini
  159. # Google Gemini API Key (Required for Brain & Vision)
  160. GEMINI_API_KEY=AIzaSy...
  161. # Groq Cloud API Key (Recommended for ultra-fast ~0.2s STT)
  162. GROQ_API_KEY=gsk_...
  163. STT_MODE=groq
  164. GROQ_WHISPER_MODEL=whisper-large-v3-turbo
  165. # Voice Listening Mode: hybrid, push_to_talk, or hi_jarvis
  166. VOICE_LISTEN_MODE=hybrid
  167. ```
  168. ### 3. Launch Jarvis
  169. ```bash
  170. python3 main.py
  171. ```
  172. ---
  173. ## ⌨️ Control & Shortcuts
  174. | Action | Shortcut / Trigger | Description |
  175. | :--- | :--- | :--- |
  176. | **Toggle Chat HUD** | `Alt + Space` | Show/hide Spotlight-style overlay chat window |
  177. | **Toggle Voice Mic** | `Cmd + Shift + J` | Press to record (Tink), press again to send (Pop) |
  178. | **Wake Word Trigger** | *"Hi Jarvis"* | Wake up assistant via speech in `hybrid` or `hi_jarvis` mode |
  179. | **Silence Cutoff** | *3.5s pause* | Automatically dispatches voice query after speaking ends |
  180. | **Switch Voice Mode** | Status Bar Menu | Change between `Hybrid`, `Push-to-Talk`, and `Hi Jarvis` |
  181. | **Inspect Work** | *"เปิดดูผลงานล่าสุด"* | Reveals completed background files/URLs on screen |
  182. ---
  183. ## 📋 Technology Stack
  184. - **Core Runtime:** Python 3.11+, PySide6 (Qt for Python 6.6+)
  185. - **Large Language Models:** Google GenAI SDK (`gemini-3.5-flash-lite`, `gemini-3.6-flash`, `gemini-3.8-flash`)
  186. - **Speech-to-Text:** Groq Cloud API (`whisper-large-v3-turbo`), Faster-Whisper (CTranslate2, int8)
  187. - **Speech Synthesis:** Microsoft Edge-TTS (`th-TH-NiwatNeural`, `en-GB-RyanNeural`), macOS `/usr/bin/afplay`
  188. - **Protocols & Standards:** Model Context Protocol (MCP) Stdio Client
  189. - **macOS Low-Level APIs:** Quartz Event Tap (`CGEventTapCreate`), ApplicationServices Accessibility, `screencapture`, `pbcopy`/`pbpaste`
  190. - **GUI Automation:** PyAutoGUI, Pillow (PIL), NumPy, SoundDevice
  191. ---
  192. ## 📄 License
  193. This project is open-sourced under the [MIT License](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.