by Tex Leave a comment Play

Superlips: dictation by mouthing the words

Tonight I came across the Subvocal, a startup building a wearable that turns silently mouthed words into text.

It works by like mapping your face and lip movements, and it seems great, but you gotta where a pair of $500 glasses (and who knows if it’ll even work well when they ship).

It got me wondering if you could get somewhere similar with the webcam you already have. Take an open lipreading model, run it on your own computer, and hand its rough guess to a small local LLM to clean up.

So I made a little open source app Superlips.

You hold down the right ⌥ option key and mouth a sentence at your webcam, and the text shows up wherever your cursor is. Everything runs on your Mac, and no video leaves the machine.

A grid of 72 grayscale crops of Tex’s mouth, one per video frame
What Superlips actually sees: my mouth, frame by frame, cut out of the webcam video and straightened so the model can read it.

To be clear, this is an experiment and not a product. It works more often than I expected, but lipreading is hard. Lips look the same for P, B and M, or F and V, so the LLM is guessing from context to fix those.

What I’ve noticed so far

  • You need good light. In a dark room it can’t see your mouth, so it doesn’t work at all. That could change if Macs get Face ID style depth cameras, or just better webcams.
  • Mouthing words feels weird. It took me a while to get used to talking to a computer. Whispering to one was harder, and silently mouthing is the hardest of all.
  • Keep your head still. It tracks your face, but moving around seems to throw it off.
  • Mouth out full sentences. The model needs context. If you only say a couple words it's going to just guess.

There’s a lot of room here

Superlips is open source, so anyone can build on it. A few directions I think could make a big difference: training it on your own face (every dictation you keep is a labeled example), mixing in a whisper-level microphone signal, and getting it off Python so it’s faster. If any of that sounds fun, fork it or send a pull request.

If you just want dictation, start here

If you’ve never tried talking to your computer instead of typing, these three are wonderful and much more polished than my experiment:

  • Handy is free, open source and runs locally.
  • Superwhisper was the original, and can run it locally, too.
  • Wispr Flow has a nice onboarding and if you've never used one of these tools I recommend starting here.

Superlips is for when you can’t make a sound, or you’re just curious how far lipreading can go.

all optional