dba78a2766
- Detection: broaden request-starters, handle contractions, allow 1-word '?' - Audio: PTT pre-roll (no clipped first word), drop Whisper silence hallucinations, greedy decoding + domain initial_prompt for faster/cleaner transcription - AI: cap spoken-answer tokens so replies return at conversational speed - Overlay: answers persist (no auto-hide); wire Ctrl+Shift+H show/hide toggle - Screen capture: freeze-frame at hotkey press (immune to focus-blur lockouts), hide selector from screen-share (NSWindowSharingNone), higher capture resolution - Stop tracking models/ and *.zip (large binaries; add to .gitignore) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
54 lines
7.0 KiB
Markdown
54 lines
7.0 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## What this is
|
||
|
||
A real-time, fully on-device AI copilot for meetings (macOS-focused). **Push-to-talk** (hold Right Option) gates audio capture; it transcribes locally with **faster-whisper** (`distil-large-v3`), detects questions, and answers via a quantized **Qwen2.5-VL-7B** vision model through **llama.cpp**. The same vision model also reads **on-screen questions**: press a hotkey, drag a box, and it answers what's in the region. Answers appear in a Qt overlay hidden from screen capture. No audio or text leaves the machine.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
./run.sh # macOS/Linux: activates .venv, sets CA bundle, installs PortAudio, runs main.py
|
||
python main.py # direct run (expects deps + models already present)
|
||
python setup.py # first-time: pip install -r requirements.txt + audio device check
|
||
run.bat # Windows launcher
|
||
```
|
||
|
||
There is **no test suite, linter, or build step**. To test the AI loop without speaking, run the app and type a question into the terminal + Enter (a background thread in `main.py` feeds terminal input straight to answer generation).
|
||
|
||
`run.sh` sets `SSL_CERT_FILE`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE` to `certs/corp_ca_bundle.pem` and `HF_HUB_DISABLE_XET=1` — this is intentional for running behind an SSL-inspecting corporate proxy. Preserve it.
|
||
|
||
## Architecture
|
||
|
||
`main.py` (`MeetingAssistant`) is the orchestrator. It wires together the components and owns **question detection** (the `_is_question` heuristic and the `_INTERROGATIVES`/`_AUX_FRONT`/`_REQUEST_STARTERS`/`_EMBEDDED_MARKERS`/`_FACT_PATTERNS`/`_NOISE_PHRASES` tables). Detection is pure regex/keyword logic — no model — and deliberately catches plain-statement questions ("I was wondering about the budget"), tag questions, math, and fact fragments.
|
||
|
||
Data flow:
|
||
1. **`src/hotkeys.py`** (`HotkeyManager`) — a pynput global listener. Holding the push-to-talk key (`audio.ptt_key`, default `alt_r` = Right Option) calls `audio_listener.arm()`/`disarm()`; the screen chord (`screen.capture_key`, default `ctrl+shift+space`) calls `overlay.request_capture()`. Callbacks run on pynput's thread, so anything touching Qt must be marshaled (see below). **Requires macOS Accessibility permission.**
|
||
2. **`src/audio_listener.py`** — one `sd.RawInputStream` per source ("microphone" / "system"). In **push-to-talk mode** (default, `audio.capture_mode`), stream callbacks buffer raw PCM *only between `arm()` and `disarm()`*; on disarm the held clip per source is queued for transcription. (The original always-on `StreamSegmenter` VAD path still exists and is used only in `continuous` mode.) A single shared `WhisperModel` (config-driven `audio.whisper_model`, int8 CPU) transcribes; calls back into `MeetingAssistant.on_audio_transcript(text, timestamp, source, speaker)`.
|
||
3. **`src/diarizer.py`** (`SpeakerDiarizer`) — the "system" loopback stream is one *mixed* channel of all remote participants, so per-utterance it computes an ECAPA-TDNN voice embedding (SpeechBrain, `models/ecapa`) and online-clusters by cosine similarity into "Person 1/2/…". Best-effort; degrades to a single "Them" if unavailable.
|
||
4. **`src/context_manager.py`** (`ContextManager`) — rolling speaker-labeled transcript (deque of last 20 utterances) used as LLM context.
|
||
5. **`src/ai_engine.py`** (`AIEngine`) — `classify()` buckets a spoken question into `math | factual | open`. **Math is solved with a safe AST evaluator (`try_solve_math`), never the LLM.** Other kinds go to llama.cpp via `create_chat_completion` (**streamed**, so a newer question interrupts an in-flight one via `interrupt_event`; all generation is serialized through `_gen_lock` since llama.cpp isn't thread-safe). The model is loaded with a `Qwen25VLChatHandler` + `mmproj` when `ai.mmproj` is present (`self.vision = True`). **`answer_from_image(path)`** sends a screenshot (base64 data-URI) + instruction as a multimodal chat message — this is the screen-question path.
|
||
6. **`src/region_capture.py`** (`RegionSelector` + `grab_region`) — fullscreen dim `QWidget` with a `QRubberBand`; on drag-release it maps to global coords, grabs the region via `mss` (scaling logical→physical by comparing mss size to the Qt screen size, so it's Retina-correct), writes a temp PNG, and fires `on_done(path)`. **Must be created on the Qt main thread.**
|
||
7. **`src/overlay.py`** (`InvisibleOverlay`) — PyQt5 always-on-top frameless window. Worker threads must update it only via the `_Bridge` Qt signals (`show_answer`/`show_status`/`capture_request`), never directly. `request_capture()` emits `capture_request`, whose slot runs `self.on_capture` (set by `main.py`) on the main thread — this is how the off-thread hotkey safely launches the selector. `_exclude_from_screen_capture()` sets macOS `NSWindowSharingNone` so screen-recorders/Zoom can't see it.
|
||
|
||
Threading model: Qt event loop on the main thread; the pynput hotkey listener, audio capture, transcription, terminal input, and each answer run on their own threads. Anything touching Qt from off-thread goes through the `_Bridge` signals. Answer generation is serialized/interruptible via `answer_lock` (main.py) + `interrupt_event` + `_gen_lock` (ai_engine). The region selector and screen answer are driven from `main.py._handle_screen_capture` → `_on_region_captured` → `_screen_answer_worker`.
|
||
|
||
### `answer_mode` (config `ai.answer_mode`)
|
||
- `auto_obvious` (default) — answer math/factual directly; show a *suggested reply* for open-ended questions.
|
||
- `auto_all` — generate a committed answer for everything.
|
||
- `suggest_only` — always a draft, never committed.
|
||
|
||
`answer_sources` (config `audio.answer_sources`) controls which speaker sources actually trigger answering vs. just adding context.
|
||
|
||
## Config & assets
|
||
|
||
- **`config.yaml`** is the live config (loaded by `main.py`). `config/settings.py` is unused legacy — ignore it.
|
||
- Models under `models/` (large binaries): `models/ecapa/` (SpeechBrain diarizer), and the **Qwen2.5-VL** vision model + `mmproj` (downloaded by `setup.py` from `ggml-org/Qwen2.5-VL-7B-Instruct-GGUF`). The Whisper STT model auto-downloads by name via faster-whisper on first run (the old committed `models/whisper/` base.en dir is now only a fallback). The old `models/Qwen2.5-7B-Instruct-Q4_K_M.gguf` (text-only) is superseded by the VL model.
|
||
- **`src/screen_scanner.py` is dead code** — superseded by `src/region_capture.py`. The OCR screen-scan was always disabled (no-op stub); screen questions now go through the draw-a-box vision path instead.
|
||
- **16 GB RAM is the binding constraint.** VL model (~4.7 GB) + mmproj (~1.4 GB) + Whisper + ECAPA + Qt ≈ 12–13 GB. If memory is tight, set `audio.whisper_model: medium.en`.
|
||
|
||
## Platform notes
|
||
|
||
macOS is the primary target. Capturing other participants requires a virtual loopback device (BlackHole) routed through a Multi-Output Device — see README "Hearing other participants". The screen-capture-hiding and loopback auto-detection (`loopback_keywords`) are core to the product, not incidental.
|