InterviewAI/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A real-time, fully on-device AI copilot for meetings (macOS-focused). Push-to-talk (hold Right Option) gates audio capture; it transcribes locally with faster-whisper (distil-large-v3), detects questions, and answers via a quantized Qwen2.5-VL-7B vision model through llama.cpp. The same vision model also reads on-screen questions: press a hotkey, drag a box, and it answers what's in the region. Answers appear in a Qt overlay hidden from screen capture. No audio or text leaves the machine.

Commands

./run.sh                 # macOS/Linux: activates .venv, sets CA bundle, installs PortAudio, runs main.py
python main.py           # direct run (expects deps + models already present)
python setup.py          # first-time: pip install -r requirements.txt + audio device check
run.bat                  # Windows launcher

There is no test suite, linter, or build step. To test the AI loop without speaking, run the app and type a question into the terminal + Enter (a background thread in main.py feeds terminal input straight to answer generation).

run.sh sets SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE to certs/corp_ca_bundle.pem and HF_HUB_DISABLE_XET=1 — this is intentional for running behind an SSL-inspecting corporate proxy. Preserve it.

Architecture

main.py (MeetingAssistant) is the orchestrator. It wires together the components and owns question detection (the _is_question heuristic and the _INTERROGATIVES/_AUX_FRONT/_REQUEST_STARTERS/_EMBEDDED_MARKERS/_FACT_PATTERNS/_NOISE_PHRASES tables). Detection is pure regex/keyword logic — no model — and deliberately catches plain-statement questions ("I was wondering about the budget"), tag questions, math, and fact fragments.

Data flow:

1. src/hotkeys.py (HotkeyManager) — a pynput global listener. Holding the push-to-talk key (audio.ptt_key, default alt_r = Right Option) calls audio_listener.arm()/disarm(); the screen chord (screen.capture_key, default ctrl+shift+space) calls overlay.request_capture(). Callbacks run on pynput's thread, so anything touching Qt must be marshaled (see below). Requires macOS Accessibility permission.
2. src/audio_listener.py — one sd.RawInputStream per source ("microphone" / "system"). In push-to-talk mode (default, audio.capture_mode), stream callbacks buffer raw PCM only between arm() and disarm(); on disarm the held clip per source is queued for transcription. (The original always-on StreamSegmenter VAD path still exists and is used only in continuous mode.) A single shared WhisperModel (config-driven audio.whisper_model, int8 CPU) transcribes; calls back into MeetingAssistant.on_audio_transcript(text, timestamp, source, speaker).
3. src/diarizer.py (SpeakerDiarizer) — the "system" loopback stream is one mixed channel of all remote participants, so per-utterance it computes an ECAPA-TDNN voice embedding (SpeechBrain, models/ecapa) and online-clusters by cosine similarity into "Person 1/2/…". Best-effort; degrades to a single "Them" if unavailable.
4. src/context_manager.py (ContextManager) — rolling speaker-labeled transcript (deque of last 20 utterances) used as LLM context.
5. src/ai_engine.py (AIEngine) — classify() buckets a spoken question into math | factual | open. Math is solved with a safe AST evaluator (try_solve_math), never the LLM. Other kinds go to llama.cpp via create_chat_completion (streamed, so a newer question interrupts an in-flight one via interrupt_event; all generation is serialized through _gen_lock since llama.cpp isn't thread-safe). The model is loaded with a Qwen25VLChatHandler + mmproj when ai.mmproj is present (self.vision = True). answer_from_image(path) sends a screenshot (base64 data-URI) + instruction as a multimodal chat message — this is the screen-question path.
6. src/region_capture.py (RegionSelector + grab_region) — fullscreen dim QWidget with a QRubberBand; on drag-release it maps to global coords, grabs the region via mss (scaling logical→physical by comparing mss size to the Qt screen size, so it's Retina-correct), writes a temp PNG, and fires on_done(path). Must be created on the Qt main thread.
7. src/overlay.py (InvisibleOverlay) — PyQt5 always-on-top frameless window. Worker threads must update it only via the _Bridge Qt signals (show_answer/show_status/capture_request), never directly. request_capture() emits capture_request, whose slot runs self.on_capture (set by main.py) on the main thread — this is how the off-thread hotkey safely launches the selector. _exclude_from_screen_capture() sets macOS NSWindowSharingNone so screen-recorders/Zoom can't see it.

Threading model: Qt event loop on the main thread; the pynput hotkey listener, audio capture, transcription, terminal input, and each answer run on their own threads. Anything touching Qt from off-thread goes through the _Bridge signals. Answer generation is serialized/interruptible via answer_lock (main.py) + interrupt_event + _gen_lock (ai_engine). The region selector and screen answer are driven from main.py._handle_screen_capture → _on_region_captured → _screen_answer_worker.

answer_mode (config ai.answer_mode)

• auto_obvious (default) — answer math/factual directly; show a suggested reply for open-ended questions.
• auto_all — generate a committed answer for everything.
• suggest_only — always a draft, never committed.

answer_sources (config audio.answer_sources) controls which speaker sources actually trigger answering vs. just adding context.

Config & assets

• config.yaml is the live config (loaded by main.py). config/settings.py is unused legacy — ignore it.
• Models under models/ (large binaries): models/ecapa/ (SpeechBrain diarizer), and the Qwen2.5-VL vision model + mmproj (downloaded by setup.py from ggml-org/Qwen2.5-VL-7B-Instruct-GGUF). The Whisper STT model auto-downloads by name via faster-whisper on first run (the old committed models/whisper/ base.en dir is now only a fallback). The old models/Qwen2.5-7B-Instruct-Q4_K_M.gguf (text-only) is superseded by the VL model.
• src/screen_scanner.py is dead code — superseded by src/region_capture.py. The OCR screen-scan was always disabled (no-op stub); screen questions now go through the draw-a-box vision path instead.
• 16 GB RAM is the binding constraint. VL model (~4.7 GB) + mmproj (~1.4 GB) + Whisper + ECAPA + Qt ≈ 12–13 GB. If memory is tight, set audio.whisper_model: medium.en.

Platform notes

macOS is the primary target. Capturing other participants requires a virtual loopback device (BlackHole) routed through a Multi-Output Device — see README "Hearing other participants". The screen-capture-hiding and loopback auto-detection (loopback_keywords) are core to the product, not incidental.