InterviewAI/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A real-time, fully on-device AI copilot for meetings (macOS-focused). Push-to-talk (hold Right Option) gates audio capture; it transcribes locally with faster-whisper (local models/whisper-base.en by default — the latency-critical choice, see Latency below), detects questions, and answers via a quantized Qwen2.5-VL-7B vision model through llama.cpp, streaming partial answers into the overlay as they generate. The same vision model also reads on-screen questions: press a hotkey, drag a box, and it answers what's in the region. Answers can be scoped to a context window (contexts/*.yaml, e.g. "Flask interview") so the model stays on-topic. Answers appear in a Qt overlay hidden from screen capture (drag interior to move, edges to resize). No audio or text leaves the machine — except the opt-in DuckDuckGo web search (ai.web_search.enabled, default off).

Commands

./run.sh                 # macOS/Linux: activates .venv, sets CA bundle, installs PortAudio, runs main.py
python main.py           # direct run (expects deps + models already present)
python setup.py          # first-time: pip install -r requirements.txt + audio device check
run.bat                  # Windows launcher

There is no test suite, linter, or build step. To test the AI loop without speaking, run the app and type a question into the terminal + Enter (a background thread in main.py feeds terminal input straight to answer generation).

run.sh sets SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE to certs/corp_ca_bundle.pem and HF_HUB_DISABLE_XET=1 — this is intentional for running behind an SSL-inspecting corporate proxy. Preserve it.

Architecture

main.py (MeetingAssistant) is the orchestrator. It wires together the components and owns question detection (the _is_question heuristic and the _INTERROGATIVES/_AUX_FRONT/_REQUEST_STARTERS/_EMBEDDED_MARKERS/_FACT_PATTERNS/_NOISE_PHRASES tables). Detection is pure regex/keyword logic — no model — and deliberately catches plain-statement questions ("I was wondering about the budget"), tag questions, math, and fact fragments.

Data flow:

1. src/hotkeys.py (HotkeyManager) — a pynput global listener. Holding the push-to-talk key (audio.ptt_key, default alt_r = Right Option) calls audio_listener.arm()/disarm(); the screen chord (screen.capture_key, default ctrl+shift+space) calls overlay.request_capture(). Callbacks run on pynput's thread, so anything touching Qt must be marshaled (see below). Requires macOS Accessibility permission.
2. src/audio_listener.py — one sd.RawInputStream per source ("microphone" / "system"). In push-to-talk mode (default, audio.capture_mode), stream callbacks buffer raw PCM only between arm() and disarm(); on disarm the held clip per source is queued for transcription. (The original always-on StreamSegmenter VAD path still exists and is used only in continuous mode; toggle_auto_listen() switches at runtime via Ctrl+Shift+M.) A single shared WhisperModel (config-driven audio.whisper_model, int8 CPU) transcribes — every transcribe() call is serialized through _model_lock (faster-whisper is not thread-safe) — then calls back into MeetingAssistant.on_audio_transcript(text, timestamp, source, speaker). The loader tries local dirs first (models/whisper-base.en → models/whisper-medium.en → …) with local_files_only for any directory, so runtime never touches Hugging Face (blocked by the corp proxy). Loopback detection (_find_system_device) walks audio.loopback_keywords in priority order — a true loopback (BlackHole) beats app-specific devices ("Microsoft Teams Audio", which is Teams-only and often silent). An optional incremental streaming transcriber (_StreamingTranscriber, audio.streaming_transcription, default OFF) transcribes the growing clip while the key is held — see Latency for why it's off.
3. src/diarizer.py (SpeakerDiarizer) — the "system" loopback stream is one mixed channel of all remote participants, so per-utterance it computes an ECAPA-TDNN voice embedding (SpeechBrain, models/ecapa) and online-clusters by cosine similarity into "Person 1/2/…". Best-effort; degrades to a single "Them" if unavailable.
4. src/context_manager.py (ContextManager) — rolling speaker-labeled transcript (deque of last 20 utterances) used as LLM context.
5. src/ai_engine.py (AIEngine) — classify() buckets a spoken question into math | factual | open. Math is solved with a safe AST evaluator (try_solve_math), never the LLM. Other kinds go to llama.cpp via create_chat_completion (streamed; all generation is serialized through _gen_lock since llama.cpp isn't thread-safe). _stream_chat emits throttled partial text via an on_partial callback (wired by main.py to the overlay with a ▌ cursor) so first words show at first-token time; it prints ⏱ gen timing per answer. Interrupt semantics: interrupt() only fires while a generation is in-flight (_generating), and the flag is cleared in a finally — a stale interrupt must never kill the next question (a real bug that dropped rapid follow-ups). _system_prompt prepends _context_preamble() — the active context window from ContextLibrary (contexts/*.yaml, hot-reloaded; strict: true = no out-of-scope variations). _maybe_web_search() (opt-in ai.web_search) grounds factual/open committed answers with DuckDuckGo snippets — never for math or suggested replies. The model is loaded with a Qwen25VLChatHandler + mmproj when ai.mmproj is present (self.vision = True). answer_from_image(path) sends a screenshot (base64 data-URI) + instruction as a multimodal chat message — this is the screen-question path (also context-scoped and partial-streamed).
6. src/region_capture.py (RegionSelector + grab_region) — fullscreen dim QWidget with a QRubberBand; on drag-release it maps to global coords, grabs the region via mss (scaling logical→physical by comparing mss size to the Qt screen size, so it's Retina-correct), writes a temp PNG, and fires on_done(path). Must be created on the Qt main thread.
7. src/overlay.py (InvisibleOverlay) — PyQt5 always-on-top frameless window. Worker threads must update it only via the _Bridge Qt signals (show_answer/show_status/capture_request), never directly. request_capture() emits capture_request, whose slot runs self.on_capture (set by main.py) on the main thread — this is how the off-thread hotkey safely launches the selector. _exclude_from_screen_capture() sets macOS NSWindowSharingNone so screen-recorders/Zoom can't see it. The text area is a _ResizeTextEdit: press near an edge/corner to resize, press the interior and drag to move. Hard-won macOS quirks baked into it: fully-transparent regions of a translucent window pass clicks through (so only the opaque widget can catch presses), and grabMouse() never delivers the drag stream for this window — drags are tracked via startSystemResize/startSystemMove first, falling back to an application-wide event filter.

Threading model: Qt event loop on the main thread; the pynput hotkey listener, audio capture, transcription, terminal input, and each answer run on their own threads. Anything touching Qt from off-thread goes through the _Bridge signals. Answer generation is serialized/interruptible via answer_lock (main.py) + interrupt_event + _gen_lock (ai_engine). The region selector and screen answer are driven from main.py._handle_screen_capture → _on_region_captured → _screen_answer_worker.

answer_mode (config ai.answer_mode)

• auto_obvious (default) — answer math/factual directly; show a suggested reply for open-ended questions.
• auto_all — generate a committed answer for everything.
• suggest_only — always a draft, never committed.

answer_sources (config audio.answer_sources) controls which speaker sources actually trigger answering vs. just adding context.

Config & assets

• config.yaml is the live config (loaded by main.py) and is gitignored — config.example.yaml is the tracked template; keep it in sync when adding config options. config/settings.py is unused legacy — ignore it.
• contexts/ holds the context-window YAMLs (see contexts/README.md). The example python_flask_backend.yaml is tracked; user-personal context files are gitignored.
• Models under models/ (large binaries, all gitignored): models/whisper-base.en/ (default STT, downloaded directly from HF via unverified urllib because the corp proxy breaks huggingface_hub), models/whisper-medium.en/ (slower/more robust alternative), models/ecapa/ (SpeechBrain diarizer), and the Qwen2.5-VL vision model + mmproj (downloaded by setup.py from ggml-org/Qwen2.5-VL-7B-Instruct-GGUF). models/whisper/ (bare model.bin, broken — missing tokenizer) and the old text-only Qwen GGUF are legacy.
• src/screen_scanner.py is dead code — superseded by src/region_capture.py. The OCR screen-scan was always disabled (no-op stub); screen questions now go through the draw-a-box vision path instead.
• 16 GB RAM is the binding constraint. VL model (~4.7 GB) + mmproj (~1.4 GB) + Whisper + ECAPA + Qt ≈ 12–13 GB.

Latency (measured, don't regress)

• faster-whisper pads every transcribe() call to a fixed 30 s encoder window, so per-call cost is nearly constant regardless of clip length (medium.en ≈ 2.3–2.9 s even for a 0.5 s tail; base.en ≈ 0.24–0.31 s). Consequences: the model choice, not pipeline tricks, dominates transcription latency, and incremental/streaming transcription (_StreamingTranscriber) barely helps on fast models while risking dropped words at chunk seams — that's why it defaults OFF.
• Generation latency is first-token (prefill) + tokens. Partial answers stream to the overlay (⚡ first words metric); the system prompt (context window) is constant per session so llama.cpp's prefix cache makes warm questions much faster than cold (~2.5 s vs ~5.6 s total observed). Keep _user_prompt lean — its transcript/memory budgets were deliberately trimmed.
• Console metrics: ⏱ whisper, ⏱ gen: first token/total, ⚡ first words on screen.

macOS permissions (frequent support issue)

All three go to the host app that launches the process (Terminal/iTerm/PyCharm), and macOS only applies them after relaunch:

• Accessibility — without it pynput logs This process is not trusted! and NO global hotkey ever fires (push-to-talk, screen grab).
• Screen & System Audio Recording — without it mss silently captures a windowless desktop (wallpaper + menu bar only), so screen questions read nothing.
• Microphone — audio capture.

Platform notes

macOS is the primary target. Capturing other participants requires a virtual loopback device (BlackHole) routed through a Multi-Output Device — see README "Hearing other participants". The screen-capture-hiding and loopback auto-detection (loopback_keywords) are core to the product, not incidental.