# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

A real-time, fully on-device AI copilot for meetings (macOS-focused). **Push-to-talk** (hold Right Option) gates audio capture; it transcribes locally with **faster-whisper** (local `models/whisper-base.en` by default — the latency-critical choice, see Latency below), detects questions, and answers via a quantized **Qwen2.5-VL-7B** vision model through **llama.cpp**, **streaming partial answers into the overlay** as they generate. The same vision model also reads **on-screen questions**: press a hotkey, drag a box, and it answers what's in the region. Answers can be scoped to a **context window** (`contexts/*.yaml`, e.g. "Flask interview") so the model stays on-topic. Answers appear in a Qt overlay hidden from screen capture (drag interior to move, edges to resize). No audio or text leaves the machine — except the **opt-in** DuckDuckGo web search (`ai.web_search.enabled`, default off).

## Commands

```bash
./run.sh                 # macOS/Linux: activates .venv, sets CA bundle, installs PortAudio, runs main.py
python main.py           # direct run (expects deps + models already present)
python setup.py          # first-time: pip install -r requirements.txt + audio device check
run.bat                  # Windows launcher
```

There is **no test suite, linter, or build step**. To test the AI loop without speaking, run the app and type a question into the terminal + Enter (a background thread in `main.py` feeds terminal input straight to answer generation).

`run.sh` sets `SSL_CERT_FILE`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE` to `certs/corp_ca_bundle.pem` and `HF_HUB_DISABLE_XET=1` — this is intentional for running behind an SSL-inspecting corporate proxy. Preserve it.

## Architecture

`main.py` (`MeetingAssistant`) is the orchestrator. It wires together the components and owns **question detection** (the `_is_question` heuristic and the `_INTERROGATIVES`/`_AUX_FRONT`/`_REQUEST_STARTERS`/`_EMBEDDED_MARKERS`/`_FACT_PATTERNS`/`_NOISE_PHRASES` tables). Detection is pure regex/keyword logic — no model — and deliberately catches plain-statement questions ("I was wondering about the budget"), tag questions, math, and fact fragments.

Data flow:
1. **`src/hotkeys.py`** (`HotkeyManager`) — a pynput global listener. Holding the push-to-talk key (`audio.ptt_key`, default `alt_r` = Right Option) calls `audio_listener.arm()`/`disarm()`; the screen chord (`screen.capture_key`, default `ctrl+shift+space`) calls `overlay.request_capture()`. Callbacks run on pynput's thread, so anything touching Qt must be marshaled (see below). **Requires macOS Accessibility permission.**
2. **`src/audio_listener.py`** — one `sd.RawInputStream` per source ("microphone" / "system"). In **push-to-talk mode** (default, `audio.capture_mode`), stream callbacks buffer raw PCM *only between `arm()` and `disarm()`*; on disarm the held clip per source is queued for transcription. (The original always-on `StreamSegmenter` VAD path still exists and is used only in `continuous` mode; `toggle_auto_listen()` switches at runtime via Ctrl+Shift+M.) A single shared `WhisperModel` (config-driven `audio.whisper_model`, int8 CPU) transcribes — **every `transcribe()` call is serialized through `_model_lock` (faster-whisper is not thread-safe)** — then calls back into `MeetingAssistant.on_audio_transcript(text, timestamp, source, speaker)`. The loader tries local dirs first (`models/whisper-base.en` → `models/whisper-medium.en` → …) with `local_files_only` for any directory, so runtime never touches Hugging Face (blocked by the corp proxy). Loopback detection (`_find_system_device`) walks `audio.loopback_keywords` in **priority order** — a true loopback (BlackHole) beats app-specific devices ("Microsoft Teams Audio", which is Teams-only and often silent). An optional **incremental streaming transcriber** (`_StreamingTranscriber`, `audio.streaming_transcription`, default OFF) transcribes the growing clip while the key is held — see Latency for why it's off.
3. **`src/diarizer.py`** (`SpeakerDiarizer`) — the "system" loopback stream is one *mixed* channel of all remote participants, so per-utterance it computes an ECAPA-TDNN voice embedding (SpeechBrain, `models/ecapa`) and online-clusters by cosine similarity into "Person 1/2/…". Best-effort; degrades to a single "Them" if unavailable.
4. **`src/context_manager.py`** (`ContextManager`) — rolling speaker-labeled transcript (deque of last 20 utterances) used as LLM context.
5. **`src/ai_engine.py`** (`AIEngine`) — `classify()` buckets a spoken question into `math | factual | open`. **Math is solved with a safe AST evaluator (`try_solve_math`), never the LLM.** Other kinds go to llama.cpp via `create_chat_completion` (**streamed**; all generation is serialized through `_gen_lock` since llama.cpp isn't thread-safe). `_stream_chat` emits throttled partial text via an `on_partial` callback (wired by `main.py` to the overlay with a `▌` cursor) so first words show at first-token time; it prints `⏱ gen` timing per answer. **Interrupt semantics:** `interrupt()` only fires while a generation is in-flight (`_generating`), and the flag is cleared in a `finally` — a stale interrupt must never kill the *next* question (a real bug that dropped rapid follow-ups). `_system_prompt` prepends `_context_preamble()` — the active **context window** from `ContextLibrary` (`contexts/*.yaml`, hot-reloaded; `strict: true` = no out-of-scope variations). `_maybe_web_search()` (opt-in `ai.web_search`) grounds factual/open committed answers with DuckDuckGo snippets — never for math or suggested replies. The model is loaded with a `Qwen25VLChatHandler` + `mmproj` when `ai.mmproj` is present (`self.vision = True`). **`answer_from_image(path)`** sends a screenshot (base64 data-URI) + instruction as a multimodal chat message — this is the screen-question path (also context-scoped and partial-streamed).
6. **`src/region_capture.py`** (`RegionSelector` + `grab_region`) — fullscreen dim `QWidget` with a `QRubberBand`; on drag-release it maps to global coords, grabs the region via `mss` (scaling logical→physical by comparing mss size to the Qt screen size, so it's Retina-correct), writes a temp PNG, and fires `on_done(path)`. **Must be created on the Qt main thread.**
7. **`src/overlay.py`** (`InvisibleOverlay`) — PyQt5 always-on-top frameless window. Worker threads must update it only via the `_Bridge` Qt signals (`show_answer`/`show_status`/`capture_request`), never directly. `request_capture()` emits `capture_request`, whose slot runs `self.on_capture` (set by `main.py`) on the main thread — this is how the off-thread hotkey safely launches the selector. `_exclude_from_screen_capture()` sets macOS `NSWindowSharingNone` so screen-recorders/Zoom can't see it. The text area is a `_ResizeTextEdit`: press near an edge/corner to **resize**, press the interior and drag to **move**. Hard-won macOS quirks baked into it: fully-transparent regions of a translucent window pass clicks through (so only the opaque widget can catch presses), and `grabMouse()` never delivers the drag stream for this window — drags are tracked via `startSystemResize`/`startSystemMove` first, falling back to an **application-wide event filter**.

Threading model: Qt event loop on the main thread; the pynput hotkey listener, audio capture, transcription, terminal input, and each answer run on their own threads. Anything touching Qt from off-thread goes through the `_Bridge` signals. Answer generation is serialized/interruptible via `answer_lock` (main.py) + `interrupt_event` + `_gen_lock` (ai_engine). The region selector and screen answer are driven from `main.py._handle_screen_capture` → `_on_region_captured` → `_screen_answer_worker`.

### `answer_mode` (config `ai.answer_mode`)
- `auto_obvious` (default) — answer math/factual directly; show a *suggested reply* for open-ended questions.
- `auto_all` — generate a committed answer for everything.
- `suggest_only` — always a draft, never committed.

`answer_sources` (config `audio.answer_sources`) controls which speaker sources actually trigger answering vs. just adding context.

## Config & assets

- **`config.yaml`** is the live config (loaded by `main.py`) and is **gitignored** — `config.example.yaml` is the tracked template; keep it in sync when adding config options. `config/settings.py` is unused legacy — ignore it.
- **`contexts/`** holds the context-window YAMLs (see `contexts/README.md`). The example `python_flask_backend.yaml` is tracked; user-personal context files are gitignored.
- Models under `models/` (large binaries, all gitignored): `models/whisper-base.en/` (default STT, downloaded directly from HF via unverified urllib because the corp proxy breaks `huggingface_hub`), `models/whisper-medium.en/` (slower/more robust alternative), `models/ecapa/` (SpeechBrain diarizer), and the **Qwen2.5-VL** vision model + `mmproj` (downloaded by `setup.py` from `ggml-org/Qwen2.5-VL-7B-Instruct-GGUF`). `models/whisper/` (bare model.bin, broken — missing tokenizer) and the old text-only Qwen GGUF are legacy.
- **`src/screen_scanner.py` is dead code** — superseded by `src/region_capture.py`. The OCR screen-scan was always disabled (no-op stub); screen questions now go through the draw-a-box vision path instead.
- **16 GB RAM is the binding constraint.** VL model (~4.7 GB) + mmproj (~1.4 GB) + Whisper + ECAPA + Qt ≈ 12–13 GB.

## Latency (measured, don't regress)

- **faster-whisper pads every `transcribe()` call to a fixed 30 s encoder window**, so per-call cost is nearly constant regardless of clip length (`medium.en` ≈ 2.3–2.9 s even for a 0.5 s tail; `base.en` ≈ 0.24–0.31 s). Consequences: the model choice, not pipeline tricks, dominates transcription latency, and incremental/streaming transcription (`_StreamingTranscriber`) barely helps on fast models while risking dropped words at chunk seams — that's why it defaults OFF.
- **Generation latency is first-token (prefill) + tokens.** Partial answers stream to the overlay (`⚡ first words` metric); the system prompt (context window) is constant per session so llama.cpp's prefix cache makes warm questions much faster than cold (~2.5 s vs ~5.6 s total observed). Keep `_user_prompt` lean — its transcript/memory budgets were deliberately trimmed.
- Console metrics: `⏱ whisper`, `⏱ gen: first token/total`, `⚡ first words on screen`.

## macOS permissions (frequent support issue)

All three go to the **host app that launches the process** (Terminal/iTerm/PyCharm), and macOS only applies them after relaunch:
- **Accessibility** — without it pynput logs `This process is not trusted!` and NO global hotkey ever fires (push-to-talk, screen grab).
- **Screen & System Audio Recording** — without it `mss` silently captures a windowless desktop (wallpaper + menu bar only), so screen questions read nothing.
- **Microphone** — audio capture.

## Platform notes

macOS is the primary target. Capturing other participants requires a virtual loopback device (BlackHole) routed through a Multi-Output Device — see README "Hearing other participants". The screen-capture-hiding and loopback auto-detection (`loopback_keywords`) are core to the product, not incidental.