# Meeting Assistant

A real-time, on-device AI copilot for meetings. You **hold a key to talk**, it
detects questions (even ones phrased as plain statements), and shows answers or
suggested replies in an overlay that is hidden from screen-capture. It can also
**read a question off your screen** when you drag a box around it.

Everything runs locally: **faster-whisper** (`distil-large-v3` by default) for
speech-to-text and a quantized **Qwen2.5-VL-7B** vision model (via `llama.cpp`)
that answers both spoken and on-screen questions. No audio or text leaves your machine.

## What it does

- **Push-to-talk — no noise.** Audio is only captured while you **hold Right
  Option (⌥)**; nothing is transcribed otherwise. Captures your microphone *and*
  the other participants' audio (system output via a loopback device), tagged by
  speaker (`You` / `Them`).
- **Reads on-screen questions.** Press **Ctrl+Shift+Space**, drag a box over a
  question (e.g. on a shared slide), and the vision model reads and answers it —
  including multiple-choice, code, and math.
- **Detects real questions** — interrogatives, yes/no questions, requests
  ("explain the rollout plan"), embedded/plain-sentence questions
  ("I was wondering about the budget"), tag questions ("…, right?"), math, and
  fact-shaped fragments ("difference between TCP and UDP").
- **Auto-fills the obvious.** Arithmetic is solved instantly (no LLM); short
  factual lookups get a direct answer. Open-ended questions get a concise
  **suggested reply** you can read out, using the live meeting transcript as context.

## Hearing other participants (one-time macOS setup)

Your microphone only captures *you*. To also capture what the other participants
say, the assistant reads your system audio output through a virtual loopback
device. **BlackHole** is already detected on this machine.

The catch: if you send audio *only* to BlackHole, you won't hear it yourself. So
create a **Multi-Output Device** that plays to both your speakers/headphones and
BlackHole at once:

1. Open **Audio MIDI Setup** (Applications → Utilities).
2. Click **+** (bottom-left) → **Create Multi-Output Device**.
3. Check both **BlackHole 2ch** and your normal output (e.g. *MacBook Pro Speakers*).
4. In **System Settings → Sound → Output**, select that Multi-Output Device.
5. In your meeting app (Zoom/Teams/Meet), make sure the speaker/output is the
   Multi-Output Device (or the system default).

Now meeting audio reaches both your ears and the assistant. Your microphone stays
selected as the meeting's *input*.

> Don't have BlackHole? Install with `brew install blackhole-2ch`, then re-run.

## Configuration (`config.yaml`)

```yaml
audio:
  source: "both"          # "microphone", "system", or "both"
  capture_mode: "push_to_talk"   # push_to_talk (hold key) or continuous
  ptt_key: "alt_r"        # push-to-talk key (Right Option). e.g. cmd_r, f8, ctrl_r
  whisper_model: "distil-large-v3"  # or large-v3, medium.en, or a local dir
  mic_device: null        # null = auto. Or a device index / name substring.
  system_device: null     # null = auto-detect loopback (BlackHole). Or index / name.
  answer_sources:         # which speakers trigger an answer
    - "system"            # other participants
    - "microphone"        # your own voice (handy for testing)

screen:
  capture_key: "ctrl+shift+space"   # press, then drag a box over a question

ai:
  model: "Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf"
  mmproj: "mmproj-Qwen2.5-VL-7B-Instruct-f16.gguf"   # required for screen reading
  answer_mode: "auto_obvious"  # auto_obvious | auto_all | suggest_only
  user_name: "you"
```

> **Tight on 16 GB RAM?** The vision model + `distil-large-v3` fit, but if memory
> gets tight set `whisper_model: medium.en`.

- `auto_obvious` — answer obvious questions (math/factual) directly; show a
  *suggested reply* for open-ended ones.
- `auto_all` — generate a full answer for every detected question.
- `suggest_only` — never commit; always show a draft.

## Run

```bash
./run.sh           # macOS / Linux
# or
python main.py
```

First run downloads the models (run `python setup.py` once, ~6 GB for the vision
model; `distil-large-v3` auto-downloads on first launch).

On first launch grant three macOS permissions (System Settings → Privacy & Security):
- **Microphone** — to capture audio.
- **Screen Recording** — so the overlay can hide *itself* from capture, and so the
  draw-a-box screen grab works.
- **Accessibility** — so the global push-to-talk and screen-grab hotkeys are seen.

You can also type a question in the terminal + Enter to test the AI directly.

## How it works

| File | Role |
|------|------|
| `src/audio_listener.py` | Captures each source. Push-to-talk buffers audio only while armed; transcribes the held clip with one shared Whisper model, tags by source. |
| `src/hotkeys.py` | Global hotkeys (pynput): hold-to-talk + screen-grab chord. |
| `main.py` | Question detection + orchestration. |
| `src/ai_engine.py` | Math fast-path, classification, and Qwen2.5-VL answering for both spoken and on-screen (`answer_from_image`) questions. |
| `src/region_capture.py` | Draw-a-box fullscreen selector + region screenshot. |
| `src/context_manager.py` | Rolling speaker-labeled meeting transcript. |
| `src/overlay.py` | Always-on-top overlay, hidden from screen capture. |
```