diff --git a/.gitignore b/.gitignore
index 57cbcfa..a7a5e72 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
node_modules/
+.opencluely-cache/
.env
.venv-whisper/
.whisper-models/
@@ -7,3 +8,11 @@ dist/
bin/
.DS_Store
*.log
+
+# Local coaching materials (not part of the application release)
+/MISSION.md
+/RESOURCES.md
+/learning-records/
+/lessons/
+/reference/
+/assets/course.css
diff --git a/README.md b/README.md
index a2ed045..6e9dcee 100644
--- a/README.md
+++ b/README.md
@@ -1,237 +1,191 @@
-# OpenCluely
+# OpenCluely — Gemini Live Fork
-**The invisible AI interview copilot.**
+**A stateful, real-time screen and conversation copilot for practice and AI-permitted calls.**
-Real-time AI help on a stealth overlay that screen sharing cannot see. Ask by voice or screenshot, and get clear answers that stream in as you need them.
-
-
- 
- 
- 
- 
-
-
-
Website |
-
Download |
-
Quick start |
-
How it works
+[What changed](#what-this-fork-changes) · [Quick start](#quick-start) · [Controls](#controls) · [Browser rail](#browser-rail) · [Upstream](https://github.com/TechyCSR/OpenCluely)
-Setup issues
+## How state is handled
-- **setup.sh will not run.** Make sure you are in the project folder (`cd OpenCluely`) and that the script is executable (`chmod +x setup.sh`). On Windows, use Git Bash.
-- **Setup stops with exit code 130.** That means Ctrl+C was pressed. Run `./setup.sh` again.
-- **Node or npm not found.** Install Node.js 18 or newer from [nodejs.org](https://nodejs.org/), restart the terminal, and retry.
+OpenCluely keeps one active-task anchor and treats brief unrelated conversation as a temporary detour. For example, a social question during a coding exercise can receive a short answer without discarding the problem statement, current approach, code, or unfinished step. An explicit task switch—or sustained work on a new task—replaces the anchor.
-
+Screen frames refine that state instead of creating a new conversation on every capture. Duplicate or low-information frames are skipped to reduce cost, latency, and answer churn.
-
-App issues
-
-- **Electron will not start or shows a blank window on Linux.** Try `npm run dev`, and make sure X11 or XWayland is available in headless setups.
-- **macOS screen capture does not work.** Grant Screen Recording permission under System Settings, Privacy and Security, then relaunch the app.
-- **Windows SmartScreen blocks the app.** Click More info, then Run anyway, or use `npm start` during development.
-- **Microphone or voice not working.** Voice is optional. For Azure, add valid keys to `.env`. For Whisper, install `openai-whisper`, `ffmpeg`, and `sox`, then set `SPEECH_PROVIDER=whisper`.
-
-
-
-
-
- Limitations
+## Configuration
-- **Screen-capture invisibility does not work on Linux.** The overlay stays hidden from screen shares and recordings only on **macOS** and **Windows**. This relies on Electron's `setContentProtection`, which maps to `NSWindowSharingNone` on macOS and `WDA_EXCLUDEFROMCAPTURE` on Windows. Electron provides **no equivalent on Linux** (neither X11 nor Wayland), so on Linux the call is a silent no-op and the overlay **will be visible** to anyone you screen-share with. This is a platform limitation, not a bug — there is no window flag on Linux that excludes a window from framebuffer capture. If you need capture-invisibility, run OpenCluely on macOS or Windows. As a partial workaround on Linux, share a single application window instead of your entire screen, or place the overlay on a monitor you are not sharing.
+The main environment values are documented in [`env.example`](env.example):
-
+| Variable | Purpose |
+| --- | --- |
+| `GOOGLE_CLOUD_PROJECT` | Billed project used for Vertex AI |
+| `GOOGLE_CLOUD_LOCATION` | Vertex AI region; defaults to `europe-west4` |
+| `GEMINI_LIVE_MODEL` | Gemini Live model resource name |
+| `SYSTEM_AUDIO_COMMAND` | Optional explicit path to `SystemAudioDump` |
+| `FFMPEG_COMMAND` | `ffmpeg` command or absolute path |
+| `CALL_COPILOT_SILENCE_MS` | Pause duration that closes a natural utterance |
+| `CALL_COPILOT_MAX_UTTERANCE_MS` | Maximum continuous utterance duration |
+| `CALL_COPILOT_VAD_FLOOR` | Minimum voice-activity energy floor |
+## Build
+```bash
+npm run build:mac
+```
-## Privacy and ethics
+The build produces Intel and Apple Silicon DMG and ZIP artifacts under `dist/`. Local builds may use an installed Apple signing identity automatically. This repository does not contain signing or notarization credentials.
-OpenCluely collects no data and sends no telemetry. Processing happens locally, and your session stays on your device. Requests to the AI provider are encrypted in transit.
+## Privacy, permissions, and responsible use
-The app is built for learning and practice. You are responsible for following the rules of any interview you take and the policies of the companies involved.
+- OpenCluely starts Live Screen and Call Copilot only after an explicit user action.
+- Screen and audio context is sent to the configured Google Cloud Vertex AI project when those features are active.
+- The app writes local diagnostic logs but does not add a telemetry service.
+- The native Electron overlay and the browser rail have different capture behavior; the browser rail is not private from screen sharing.
+- Record or process a conversation only with appropriate consent and where recording and AI assistance are allowed.
+- Use the project for learning, accessibility, personal productivity, and disclosed assistance—not to violate interview, assessment, workplace, or platform rules.
-## License
+## Current limitations
-Released under the MIT License. See [LICENSE](LICENSE) for details.
+- Gemini Live system-audio capture is currently implemented for macOS.
+- The browser rail must be reactivated after navigation.
+- Builds are not notarized by this fork.
+- Screen-capture exclusion depends on the operating system and capture software and should not be treated as a security boundary.
+- Gemini Live model availability, quotas, and regions depend on your Google Cloud account.
-## Acknowledgments
+## Validation performed
-- Google Gemini for the AI reasoning
-- Azure Speech and OpenAI Whisper for optional voice input
-- Electron for the cross platform desktop runtime
-- [Vysper by varun-singhh](https://github.com/varun-singhh/Vysper) for UI and structure inspiration
+- JavaScript, shell, JSON, and whitespace checks
+- Browser bridge test: extension origin accepted, webpage origin rejected
+- Credential and machine-specific identifier scan
+- Packaged ASAR inspection confirming legacy Whisper runtime and local coaching artifacts are excluded
+- Successful macOS x64 and arm64 DMG/ZIP builds
-
+## Upstream and license
-Built by [TechyCSR](https://techycsr.dev). If OpenCluely helped you, consider giving it a star ⭐
+This project is a fork of [TechyCSR/OpenCluely](https://github.com/TechyCSR/OpenCluely). The upstream project, its authors, and prior contributors remain credited in Git history. New fork-specific work is proposed upstream separately where appropriate.
-
+Released under the [Apache License 2.0](LICENSE).
diff --git a/browser-extension/background.js b/browser-extension/background.js
new file mode 100644
index 0000000..990992e
--- /dev/null
+++ b/browser-extension/background.js
@@ -0,0 +1,95 @@
+const WS_URL = 'ws://127.0.0.1:17321/?token=oc_local_7f86c1d293b74b5eb8ae3fd4';
+const ALLOWED_COMMANDS = new Set([
+ 'toggle-live-screen',
+ 'toggle-call-copilot',
+ 'send-chat-message',
+ 'get-status'
+]);
+
+const railTabs = new Set();
+let socket;
+let reconnectTimer;
+let requestSequence = 0;
+
+const broadcastToRails = message => {
+ railTabs.forEach(tabId => {
+ chrome.tabs.sendMessage(tabId, message).catch(() => railTabs.delete(tabId));
+ });
+};
+
+const setConnection = connected => {
+ broadcastToRails({ type: 'bridge-connection', connected });
+};
+
+const sendCommand = (command, payload = {}) => {
+ if (!ALLOWED_COMMANDS.has(command) || socket?.readyState !== WebSocket.OPEN) return false;
+ socket.send(JSON.stringify({
+ type: 'command',
+ command,
+ payload,
+ requestId: String(++requestSequence)
+ }));
+ return true;
+};
+
+const scheduleReconnect = () => {
+ clearTimeout(reconnectTimer);
+ if (railTabs.size) reconnectTimer = setTimeout(connect, 1500);
+};
+
+const connect = () => {
+ if (socket?.readyState === WebSocket.OPEN || socket?.readyState === WebSocket.CONNECTING) return;
+ clearTimeout(reconnectTimer);
+ try {
+ socket = new WebSocket(WS_URL);
+ } catch (_) {
+ scheduleReconnect();
+ return;
+ }
+ socket.addEventListener('open', () => {
+ setConnection(true);
+ sendCommand('get-status');
+ });
+ socket.addEventListener('message', event => {
+ try {
+ broadcastToRails({ type: 'bridge-message', message: JSON.parse(event.data) });
+ } catch (_) {}
+ });
+ socket.addEventListener('close', () => {
+ setConnection(false);
+ scheduleReconnect();
+ });
+ socket.addEventListener('error', () => socket.close());
+};
+
+chrome.action.onClicked.addListener(async tab => {
+ if (!tab.id) return;
+ railTabs.add(tab.id);
+ try {
+ await chrome.scripting.executeScript({ target: { tabId: tab.id }, files: ['content.js'] });
+ connect();
+ } catch (_) {
+ railTabs.delete(tab.id);
+ await chrome.action.setBadgeText({ tabId: tab.id, text: 'ERR' });
+ await chrome.action.setBadgeBackgroundColor({ tabId: tab.id, color: '#b42318' });
+ }
+});
+
+chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
+ if (message?.type === 'rail-ready' && sender.tab?.id) {
+ railTabs.add(sender.tab.id);
+ connect();
+ sendResponse({ connected: socket?.readyState === WebSocket.OPEN });
+ return false;
+ }
+ if (message?.type === 'bridge-command') {
+ sendResponse({ sent: sendCommand(message.command, message.payload || {}) });
+ return false;
+ }
+ return false;
+});
+
+chrome.tabs.onRemoved.addListener(tabId => railTabs.delete(tabId));
+chrome.tabs.onUpdated.addListener((tabId, changeInfo) => {
+ if (changeInfo.status === 'loading') railTabs.delete(tabId);
+});
diff --git a/browser-extension/content.js b/browser-extension/content.js
new file mode 100644
index 0000000..8e524ff
--- /dev/null
+++ b/browser-extension/content.js
@@ -0,0 +1,286 @@
+(() => {
+ if (document.getElementById('opencluely-browser-rail-host')) return;
+
+ const host = document.createElement('div');
+ host.id = 'opencluely-browser-rail-host';
+ host.style.cssText = 'all:initial;position:fixed;inset:0 0 auto auto;z-index:2147483647;pointer-events:none;';
+ document.documentElement.appendChild(host);
+ const root = host.attachShadow({ mode: 'open' });
+
+ root.innerHTML = `
+
+ `;
+
+ const rail = root.querySelector('.rail');
+ const mark = root.querySelector('.mark');
+ const state = root.querySelector('.state');
+ const answer = root.querySelector('.answer');
+ const intent = root.querySelector('.intent-text');
+ const collapse = root.querySelector('.collapse');
+ const liveScreen = root.querySelector('.live-screen');
+ const liveAudio = root.querySelector('.live-audio');
+ const input = root.querySelector('textarea');
+ const send = root.querySelector('.send');
+
+ let currentText = '';
+
+ const setConnection = connected => {
+ mark.classList.toggle('connected', connected);
+ state.textContent = connected ? 'connected' : 'offline';
+ };
+
+ const setAnswer = text => {
+ currentText = String(text || '');
+ answer.textContent = currentText || 'Waiting for an answer…';
+ answer.classList.toggle('empty', !currentText);
+ };
+
+ const deriveIntent = text => {
+ const clean = String(text || '').replace(/\*\*/g, '');
+ const match = clean.match(/(?:^|\n)Intent\s*[—:-]\s*([^\n]+)/i);
+ if (match) intent.textContent = match[1].trim();
+ };
+
+ const command = (name, payload = {}) => {
+ chrome.runtime.sendMessage({ type: 'bridge-command', command: name, payload });
+ };
+
+ const handleEvent = ({ channel, data = {} }) => {
+ if (channel === 'transcription-llm-response-start') {
+ setAnswer('');
+ rail.classList.remove('collapsed');
+ collapse.textContent = '›';
+ } else if (channel === 'transcription-llm-response-chunk') {
+ setAnswer(currentText + String(data.delta || ''));
+ deriveIntent(currentText);
+ answer.scrollTop = answer.scrollHeight;
+ } else if (channel === 'transcription-llm-response') {
+ setAnswer(data.response || data.content || currentText);
+ deriveIntent(currentText);
+ } else if (channel === 'llm-response') {
+ setAnswer(data.response || data.content || '');
+ deriveIntent(currentText);
+ } else if (channel === 'live-screen-status') {
+ liveScreen.classList.toggle('active', !!data.active);
+ liveScreen.textContent = data.active ? 'Screen active' : 'Live Screen';
+ } else if (channel === 'call-copilot-status') {
+ const active = !!(data.isCapturing || data.liveConnected);
+ liveAudio.classList.toggle('active', active);
+ liveAudio.textContent = active ? 'Audio active' : 'Live Audio';
+ } else if (channel === 'llm-error' || channel === 'ocr-error') {
+ setAnswer(`OpenCluely error: ${data.error || 'Unknown error'}`);
+ } else if (channel === 'session-cleared') {
+ intent.textContent = 'Waiting for screen context…';
+ setAnswer('');
+ }
+ };
+
+ const handleBridgeMessage = message => {
+ if (message.type === 'event') handleEvent(message);
+ if (message.type === 'command-result' && message.result?.status) {
+ const statusResult = message.result.status;
+ handleEvent({ channel: 'live-screen-status', data: statusResult.liveScreen || {} });
+ handleEvent({ channel: 'call-copilot-status', data: statusResult.callCopilot || {} });
+ }
+ };
+
+ const setCollapsed = collapsed => {
+ rail.classList.toggle('collapsed', collapsed);
+ collapse.textContent = collapsed ? '‹' : '›';
+ collapse.setAttribute('aria-expanded', String(!collapsed));
+ collapse.setAttribute('aria-label', collapsed ? 'Expand OpenCluely' : 'Collapse OpenCluely');
+ collapse.title = collapsed ? 'Expand' : 'Collapse';
+ };
+ collapse.addEventListener('click', () => setCollapsed(!rail.classList.contains('collapsed')));
+ liveScreen.addEventListener('click', () => command('toggle-live-screen'));
+ liveAudio.addEventListener('click', () => command('toggle-call-copilot'));
+ const sendMessage = () => {
+ const text = input.value.trim();
+ if (!text) return;
+ command('send-chat-message', { text });
+ input.value = '';
+ };
+ send.addEventListener('click', sendMessage);
+ input.addEventListener('keydown', event => {
+ if (event.key === 'Enter' && !event.shiftKey) {
+ event.preventDefault();
+ sendMessage();
+ }
+ });
+ root.addEventListener('click', event => event.stopPropagation());
+ root.addEventListener('keydown', event => event.stopPropagation());
+ chrome.runtime.onMessage.addListener(message => {
+ if (message?.type === 'bridge-connection') setConnection(!!message.connected);
+ if (message?.type === 'bridge-message' && message.message) handleBridgeMessage(message.message);
+ });
+ chrome.runtime.sendMessage({ type: 'rail-ready' }, response => {
+ setConnection(!!response?.connected);
+ });
+})();
diff --git a/browser-extension/manifest.json b/browser-extension/manifest.json
new file mode 100644
index 0000000..0f25e5f
--- /dev/null
+++ b/browser-extension/manifest.json
@@ -0,0 +1,19 @@
+{
+ "manifest_version": 3,
+ "name": "OpenCluely Browser Rail",
+ "version": "1.1.0",
+ "description": "Shows local OpenCluely responses in the browser tab you explicitly activate.",
+ "permissions": [
+ "activeTab",
+ "scripting"
+ ],
+ "host_permissions": [
+ "http://127.0.0.1/*"
+ ],
+ "background": {
+ "service_worker": "background.js"
+ },
+ "action": {
+ "default_title": "Show OpenCluely rail in this tab"
+ }
+}
diff --git a/chat.html b/chat.html
index b0f68c4..4e52ad2 100644
--- a/chat.html
+++ b/chat.html
@@ -14,8 +14,9 @@
margin: 0;
padding: 0;
overflow: hidden;
- font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto,
- sans-serif;
+ font-family: "Avenir Next", "Helvetica Neue", sans-serif;
+ -webkit-font-smoothing: antialiased;
+ text-rendering: optimizeLegibility;
}
.chat-container {
@@ -26,7 +27,7 @@
rgba(0, 0, 0, 0.3) 0%,
rgba(20, 20, 20, 0.4) 100%
);
- backdrop-filter: blur(25px);
+ backdrop-filter: none;
border-radius: 12px;
border: 1px solid rgba(255, 255, 255, 0.1);
box-shadow: 0 4px 25px rgba(0, 0, 0, 0.15);
@@ -42,7 +43,7 @@
justify-content: space-between;
-webkit-app-region: drag;
background: rgba(0, 0, 0, 0.2);
- backdrop-filter: blur(10px);
+ backdrop-filter: none;
cursor: move;
flex-shrink: 0;
}
@@ -80,7 +81,7 @@
height: 8px;
border-radius: 50%;
background: #ff4757;
- animation: pulse 2s infinite;
+ animation: none;
display: none;
box-shadow: 0 0 10px rgba(255, 71, 87, 0.5);
}
@@ -161,8 +162,15 @@
background: rgba(255, 152, 0, 0.1);
}
.message.assistant {
- background: rgba(156, 39, 176, 0.1);
- border-left: 3px solid #9c27b0;
+ background: linear-gradient(145deg, rgba(31, 44, 38, 0.92), rgba(18, 23, 21, 0.92));
+ border: 1px solid rgba(142, 227, 179, 0.16);
+ border-left: 4px solid #8ee3b3;
+ border-radius: 10px;
+ padding: 16px 18px;
+ color: #f6f7f5;
+ font-size: 17px;
+ line-height: 1.55;
+ letter-spacing: -0.01em;
/* Removed all height restrictions and overflow hidden */
display: block;
}
@@ -173,6 +181,8 @@
word-wrap: break-word;
word-break: break-word;
overflow-wrap: break-word;
+ font-size: 17px;
+ line-height: 1.55;
}
/* Ensure all content in assistant messages is fully visible */
@@ -323,12 +333,12 @@
color: rgba(255, 255, 255, 1);
}
- .message.assistant h1 { font-size: 16px; }
- .message.assistant h2 { font-size: 15px; }
- .message.assistant h3 { font-size: 14px; }
+ .message.assistant h1 { font-size: 24px; }
+ .message.assistant h2 { font-size: 21px; }
+ .message.assistant h3 { font-size: 19px; }
.message.assistant h4,
.message.assistant h5,
- .message.assistant h6 { font-size: 13px; }
+ .message.assistant h6 { font-size: 17px; }
/* Ensure paragraphs are fully visible */
.message.assistant p {
diff --git a/env.example b/env.example
index d770c1e..590865c 100644
--- a/env.example
+++ b/env.example
@@ -1,31 +1,18 @@
-# Google Gemini API Configuration
-# Get your API key from: https://makersuite.google.com/app/apikey
-GEMINI_API_KEY=your_gemini_api_key_here
+# Text LLM provider: antigravity (Google OAuth via agy CLI) or gemini (Vertex AI)
+LLM_PROVIDER=gemini
+ANTIGRAVITY_COMMAND=
+ANTIGRAVITY_MODEL=Gemini 3.1 Pro (Low)
+ANTIGRAVITY_TIMEOUT_MS=120000
-# Speech Recognition Configuration
-# Choose one provider: azure or whisper
-SPEECH_PROVIDER=whisper
+# Vertex AI and Gemini Live. Authenticate once with:
+# gcloud auth application-default login
+GOOGLE_CLOUD_PROJECT=
+GOOGLE_CLOUD_LOCATION=europe-west4
+GEMINI_LIVE_MODEL=gemini-live-2.5-flash-native-audio
-# Optional: Azure Speech Services Configuration
-AZURE_SPEECH_KEY=your_azure_speech_key_here
-AZURE_SPEECH_REGION=your_azure_region_here
-
-# Optional: Local OpenAI Whisper Configuration
-# Requires a local Whisper CLI installation, for example:
-# pip install openai-whisper
-# brew install ffmpeg sox
-# Use `whisper`, `python3 -m whisper`, or on Windows `.venv-whisper/Scripts/whisper.exe`
-WHISPER_COMMAND=whisper
-# Optional: where Whisper model weights are stored. Leave unset to use a stable
-# app-data folder (recommended). Set an ABSOLUTE path to override; a relative
-# path is ignored because it cannot be resolved reliably in packaged builds.
-# WHISPER_MODEL_DIR=
-WHISPER_MODEL=small
-WHISPER_LANGUAGE=auto
-WHISPER_SEGMENT_MS=4000
-WHISPER_DEVICE=auto
-WHISPER_PYTHON=
-WHISPER_CAPTURE_MODE=vad
-WHISPER_RESPONSE_TARGET=both
-WHISPER_MANUAL_MAX_MS=90000
-WHISPER_GPU_IDLE_MS=60000
+# macOS Call Copilot (system audio -> Gemini Live)
+SYSTEM_AUDIO_COMMAND=
+FFMPEG_COMMAND=ffmpeg
+CALL_COPILOT_SILENCE_MS=800
+CALL_COPILOT_MAX_UTTERANCE_MS=20000
+CALL_COPILOT_VAD_FLOOR=0.006
diff --git a/index.html b/index.html
index 31e37f5..a8d2844 100644
--- a/index.html
+++ b/index.html
@@ -22,7 +22,7 @@
rgba(0, 0, 0, 0.4) 0%,
rgba(20, 20, 20, 0.5) 100%
);
- backdrop-filter: blur(20px);
+ backdrop-filter: none;
border-radius: 8px;
display: flex;
align-items: center;
@@ -73,7 +73,7 @@
.command-item.recording i {
color: #ff4757;
text-shadow: 0 0 10px rgba(255, 71, 87, 0.5);
- animation: pulse 2s infinite;
+ animation: none;
}
.command-item.active {
@@ -103,6 +103,32 @@
box-shadow: 0 0 0 1px rgba(96, 165, 250, 0.25) inset;
}
+ #exitButton {
+ width: 26px;
+ height: 26px;
+ border-radius: 6px;
+ display: inline-flex;
+ align-items: center;
+ justify-content: center;
+ padding: 0;
+ color: rgba(248, 113, 113, 0.82);
+ }
+
+ #exitButton i {
+ font-size: 13px;
+ }
+
+ #exitButton:hover {
+ background: rgba(239, 68, 68, 0.18);
+ color: #fca5a5;
+ box-shadow: 0 0 0 1px rgba(248, 113, 113, 0.28) inset;
+ }
+
+ #exitButton:focus-visible {
+ outline: 2px solid rgba(248, 113, 113, 0.72);
+ outline-offset: 2px;
+ }
+
.command-separator {
width: 1px;
height: 16px;
@@ -127,7 +153,7 @@
.status-dot.interactive {
background-color: #10b981; /* Green for active/interactive */
box-shadow: 0 0 10px rgba(16, 185, 129, 0.6);
- animation: pulse-green 2s infinite;
+ animation: none;
}
.status-dot.non-interactive {
@@ -263,7 +289,7 @@
rgba(20, 20, 20, 0.7) 0%,
rgba(10, 10, 10, 0.6) 100%
);
- backdrop-filter: blur(18px);
+ backdrop-filter: none;
border: 1px solid rgba(255, 255, 255, 0.12);
border-radius: 10px;
color: rgba(255, 255, 255, 0.95);
@@ -358,30 +384,34 @@
⌘⇧S
-