Meeting Assistant

A real-time, on-device AI copilot for meetings. You hold a key to talk, it detects questions (even ones phrased as plain statements), and shows answers or suggested replies in an overlay that is hidden from screen-capture. It can also read a question off your screen when you drag a box around it.

Everything runs locally: faster-whisper (distil-large-v3 by default) for speech-to-text and a quantized Qwen2.5-VL-7B vision model (via llama.cpp) that answers both spoken and on-screen questions. No audio or text leaves your machine.

What it does

• Push-to-talk — no noise. Audio is only captured while you hold Right Option (⌥); nothing is transcribed otherwise. Captures your microphone and the other participants' audio (system output via a loopback device), tagged by speaker (You / Them).
• Reads on-screen questions. Press Ctrl+Shift+Space, drag a box over a question (e.g. on a shared slide), and the vision model reads and answers it — including multiple-choice, code, and math.
• Detects real questions — interrogatives, yes/no questions, requests ("explain the rollout plan"), embedded/plain-sentence questions ("I was wondering about the budget"), tag questions ("…, right?"), math, and fact-shaped fragments ("difference between TCP and UDP").
• Auto-fills the obvious. Arithmetic is solved instantly (no LLM); short factual lookups get a direct answer. Open-ended questions get a concise suggested reply you can read out, using the live meeting transcript as context.

Hearing other participants (one-time macOS setup)

Your microphone only captures you. To also capture what the other participants say, the assistant reads your system audio output through a virtual loopback device. BlackHole is already detected on this machine.

The catch: if you send audio only to BlackHole, you won't hear it yourself. So create a Multi-Output Device that plays to both your speakers/headphones and BlackHole at once:

1. Open Audio MIDI Setup (Applications → Utilities).
2. Click + (bottom-left) → Create Multi-Output Device.
3. Check both BlackHole 2ch and your normal output (e.g. MacBook Pro Speakers).
4. In System Settings → Sound → Output, select that Multi-Output Device.
5. In your meeting app (Zoom/Teams/Meet), make sure the speaker/output is the Multi-Output Device (or the system default).

Now meeting audio reaches both your ears and the assistant. Your microphone stays selected as the meeting's input.

Don't have BlackHole? Install with brew install blackhole-2ch, then re-run.

Configuration (config.yaml)

audio:
  source: "both"          # "microphone", "system", or "both"
  capture_mode: "push_to_talk"   # push_to_talk (hold key) or continuous
  ptt_key: "alt_r"        # push-to-talk key (Right Option). e.g. cmd_r, f8, ctrl_r
  whisper_model: "distil-large-v3"  # or large-v3, medium.en, or a local dir
  mic_device: null        # null = auto. Or a device index / name substring.
  system_device: null     # null = auto-detect loopback (BlackHole). Or index / name.
  answer_sources:         # which speakers trigger an answer
    - "system"            # other participants
    - "microphone"        # your own voice (handy for testing)

screen:
  capture_key: "ctrl+shift+space"   # press, then drag a box over a question

ai:
  model: "Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf"
  mmproj: "mmproj-Qwen2.5-VL-7B-Instruct-f16.gguf"   # required for screen reading
  answer_mode: "auto_obvious"  # auto_obvious | auto_all | suggest_only
  user_name: "you"

Tight on 16 GB RAM? The vision model + distil-large-v3 fit, but if memory gets tight set whisper_model: medium.en.

• auto_obvious — answer obvious questions (math/factual) directly; show a suggested reply for open-ended ones.
• auto_all — generate a full answer for every detected question.
• suggest_only — never commit; always show a draft.

Run

./run.sh           # macOS / Linux
# or
python main.py

First run downloads the models (run python setup.py once, ~6 GB for the vision model; distil-large-v3 auto-downloads on first launch).

On first launch grant three macOS permissions (System Settings → Privacy & Security):

• Microphone — to capture audio.
• Screen Recording — so the overlay can hide itself from capture, and so the draw-a-box screen grab works.
• Accessibility — so the global push-to-talk and screen-grab hotkeys are seen.

You can also type a question in the terminal + Enter to test the AI directly.

How it works

File	Role
src/audio_listener.py	Captures each source. Push-to-talk buffers audio only while armed; transcribes the held clip with one shared Whisper model, tags by source.
src/hotkeys.py	Global hotkeys (pynput): hold-to-talk + screen-grab chord.
main.py	Question detection + orchestration.
src/ai_engine.py	Math fast-path, classification, and Qwen2.5-VL answering for both spoken and on-screen (answer_from_image) questions.
src/region_capture.py	Draw-a-box fullscreen selector + region screenshot.
src/context_manager.py	Rolling speaker-labeled meeting transcript.
src/overlay.py	Always-on-top overlay, hidden from screen capture.