Improve question detection, capture stealth, latency; stop tracking model binaries
- Detection: broaden request-starters, handle contractions, allow 1-word '?' - Audio: PTT pre-roll (no clipped first word), drop Whisper silence hallucinations, greedy decoding + domain initial_prompt for faster/cleaner transcription - AI: cap spoken-answer tokens so replies return at conversational speed - Overlay: answers persist (no auto-hide); wire Ctrl+Shift+H show/hide toggle - Screen capture: freeze-frame at hotkey press (immune to focus-blur lockouts), hide selector from screen-share (NSWindowSharingNone), higher capture resolution - Stop tracking models/ and *.zip (large binaries; add to .gitignore) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Charles Wambua
113
README.md
Normal file
113
README.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Meeting Assistant
|
||||
|
||||
A real-time, on-device AI copilot for meetings. You **hold a key to talk**, it
|
||||
detects questions (even ones phrased as plain statements), and shows answers or
|
||||
suggested replies in an overlay that is hidden from screen-capture. It can also
|
||||
**read a question off your screen** when you drag a box around it.
|
||||
|
||||
Everything runs locally: **faster-whisper** (`distil-large-v3` by default) for
|
||||
speech-to-text and a quantized **Qwen2.5-VL-7B** vision model (via `llama.cpp`)
|
||||
that answers both spoken and on-screen questions. No audio or text leaves your machine.
|
||||
|
||||
## What it does
|
||||
|
||||
- **Push-to-talk — no noise.** Audio is only captured while you **hold Right
|
||||
Option (⌥)**; nothing is transcribed otherwise. Captures your microphone *and*
|
||||
the other participants' audio (system output via a loopback device), tagged by
|
||||
speaker (`You` / `Them`).
|
||||
- **Reads on-screen questions.** Press **Ctrl+Shift+Space**, drag a box over a
|
||||
question (e.g. on a shared slide), and the vision model reads and answers it —
|
||||
including multiple-choice, code, and math.
|
||||
- **Detects real questions** — interrogatives, yes/no questions, requests
|
||||
("explain the rollout plan"), embedded/plain-sentence questions
|
||||
("I was wondering about the budget"), tag questions ("…, right?"), math, and
|
||||
fact-shaped fragments ("difference between TCP and UDP").
|
||||
- **Auto-fills the obvious.** Arithmetic is solved instantly (no LLM); short
|
||||
factual lookups get a direct answer. Open-ended questions get a concise
|
||||
**suggested reply** you can read out, using the live meeting transcript as context.
|
||||
|
||||
## Hearing other participants (one-time macOS setup)
|
||||
|
||||
Your microphone only captures *you*. To also capture what the other participants
|
||||
say, the assistant reads your system audio output through a virtual loopback
|
||||
device. **BlackHole** is already detected on this machine.
|
||||
|
||||
The catch: if you send audio *only* to BlackHole, you won't hear it yourself. So
|
||||
create a **Multi-Output Device** that plays to both your speakers/headphones and
|
||||
BlackHole at once:
|
||||
|
||||
1. Open **Audio MIDI Setup** (Applications → Utilities).
|
||||
2. Click **+** (bottom-left) → **Create Multi-Output Device**.
|
||||
3. Check both **BlackHole 2ch** and your normal output (e.g. *MacBook Pro Speakers*).
|
||||
4. In **System Settings → Sound → Output**, select that Multi-Output Device.
|
||||
5. In your meeting app (Zoom/Teams/Meet), make sure the speaker/output is the
|
||||
Multi-Output Device (or the system default).
|
||||
|
||||
Now meeting audio reaches both your ears and the assistant. Your microphone stays
|
||||
selected as the meeting's *input*.
|
||||
|
||||
> Don't have BlackHole? Install with `brew install blackhole-2ch`, then re-run.
|
||||
|
||||
## Configuration (`config.yaml`)
|
||||
|
||||
```yaml
|
||||
audio:
|
||||
source: "both" # "microphone", "system", or "both"
|
||||
capture_mode: "push_to_talk" # push_to_talk (hold key) or continuous
|
||||
ptt_key: "alt_r" # push-to-talk key (Right Option). e.g. cmd_r, f8, ctrl_r
|
||||
whisper_model: "distil-large-v3" # or large-v3, medium.en, or a local dir
|
||||
mic_device: null # null = auto. Or a device index / name substring.
|
||||
system_device: null # null = auto-detect loopback (BlackHole). Or index / name.
|
||||
answer_sources: # which speakers trigger an answer
|
||||
- "system" # other participants
|
||||
- "microphone" # your own voice (handy for testing)
|
||||
|
||||
screen:
|
||||
capture_key: "ctrl+shift+space" # press, then drag a box over a question
|
||||
|
||||
ai:
|
||||
model: "Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf"
|
||||
mmproj: "mmproj-Qwen2.5-VL-7B-Instruct-f16.gguf" # required for screen reading
|
||||
answer_mode: "auto_obvious" # auto_obvious | auto_all | suggest_only
|
||||
user_name: "you"
|
||||
```
|
||||
|
||||
> **Tight on 16 GB RAM?** The vision model + `distil-large-v3` fit, but if memory
|
||||
> gets tight set `whisper_model: medium.en`.
|
||||
|
||||
- `auto_obvious` — answer obvious questions (math/factual) directly; show a
|
||||
*suggested reply* for open-ended ones.
|
||||
- `auto_all` — generate a full answer for every detected question.
|
||||
- `suggest_only` — never commit; always show a draft.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
./run.sh # macOS / Linux
|
||||
# or
|
||||
python main.py
|
||||
```
|
||||
|
||||
First run downloads the models (run `python setup.py` once, ~6 GB for the vision
|
||||
model; `distil-large-v3` auto-downloads on first launch).
|
||||
|
||||
On first launch grant three macOS permissions (System Settings → Privacy & Security):
|
||||
- **Microphone** — to capture audio.
|
||||
- **Screen Recording** — so the overlay can hide *itself* from capture, and so the
|
||||
draw-a-box screen grab works.
|
||||
- **Accessibility** — so the global push-to-talk and screen-grab hotkeys are seen.
|
||||
|
||||
You can also type a question in the terminal + Enter to test the AI directly.
|
||||
|
||||
## How it works
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `src/audio_listener.py` | Captures each source. Push-to-talk buffers audio only while armed; transcribes the held clip with one shared Whisper model, tags by source. |
|
||||
| `src/hotkeys.py` | Global hotkeys (pynput): hold-to-talk + screen-grab chord. |
|
||||
| `main.py` | Question detection + orchestration. |
|
||||
| `src/ai_engine.py` | Math fast-path, classification, and Qwen2.5-VL answering for both spoken and on-screen (`answer_from_image`) questions. |
|
||||
| `src/region_capture.py` | Draw-a-box fullscreen selector + region screenshot. |
|
||||
| `src/context_manager.py` | Rolling speaker-labeled meeting transcript. |
|
||||
| `src/overlay.py` | Always-on-top overlay, hidden from screen capture. |
|
||||
```
|
||||
Reference in New Issue
Block a user