#speech-to-text

• Speech-to-text. Cannot stress this enough but talking to the machine means you’re more likely to share more about what you want it to do.
• I maintain some basic prompts and context for copy-pasting at the end or the beginning of what I entered.
• I ended up preloading executables on the PATH that override the default ones, steering Claude toward the right tools, e.g. running python asks it to use uv.
• I use the task tool frequently for basic parallelization and context isolation.
• Simply taking time to talk to the machine and give clear instructions outperforms elaborate pre-written prompts.
• Forcing myself to evaluate the automation has another benefit: I’m less likely to just blindly assume it helps me.

• Gemini 2.5 Flash Preview TTS ($0.50/1 M input, $10.00/1 M output): ~$0.8 per hour
• GPT-4o-mini-TTS ($0.60/1 M input, $12.00/1 M output): ~$0.9/hour
• Gemini 2.5 Pro Preview TTS ($1.00/1 M input, $20.00/1 M output): ~$1.5 per hour
• GPT-4o-TTS (known as gpt-4o-audio-preview, $2.50/1 M input, $80/1 M output): ~$6.0/hour
• This is comparable to the earlier OpenAI Standard TTS ($0.75), OpenAI HD TTS ($1.5), Google Neural2 ($0.8). ElevenLabs Pro costs ~$6/hr.

• Nomic Embed Multimodal models are the current SOTA on multi-modal embeddings. Notably, they embed PDFs natively.
• Hailuo Speech-02 is the best speech model right now beating ElevenLabs. It has excellent voice cloning. Pricing: $30/1M chars. 10% of ElevenLabs, 2X of OpenAI TTS
• PaperBench is an open testing framework from OpenAI that requires models to replicate the research work in papers. It has ~8,000 tasks evaluated by LLMs and with LLMs judging the judges as well. The code is well worth studying.
• Runway Gen 4 was released with very high character consistency and longer durations
• Dreamina creates lip-synced videos from audio + a single image. Hedra is better for animated characters, though.
• Meta shared but has not released Mocha, an open character generation model that generates new characters speaking based on an audio you provide. It is not based on existing images but the quality is very good
• All Hands has a free online version where you can fix GitHub issues.

• Personal writing with connection won't go away. AI can't give you heartbreak. But the rest of non fiction writing will vanish.
• What AI is extraordinary at is personalizing to each audience member's interest
• Outlier opinions will thrive among humans - since AI is trained on consensus.
• Managers tend to be good at working with LLMs because it's mostly about delegation.
• LLMs are perfect for things that don't have a wrong answer! -- Benedict Evans.
• 💡 Explore arguing with AI. It's a safe way to get into a confrontational emotional state (which has its own benefits.)
• 💡 Keep an LLM on in voice mode while reading and ask it any questions you have.
• What models are good for what?
  • GPT 4.5 is great for creation - has a great sense of humor but a corporate style. Still, way better than GPT 4o.
  • ChatGPT is good for voice transcription and note taking. (Increasingly we take notes for AI rather than ourselves.)
  • Claude 3.7 has the best style of writing. It's also great for drawing charts.
  • O1 Pro and Deep Research is great for consumption - research.
  • Grok is the least corporate, able to argue with you, and the latest knowledge cutoff.
  • ElevenLabs for editing podcasts in your voice, making corrections.

• Deepgram has a MediaRecorder API, which is perfect.
• Whisper Streaming Web is a web app that can transcribe audio real-time from the browser. A good approach, but I wouldn't use it for meeting transcription on my mid-end laptop. Streaming takes up the bulk of my GPU, leaving little for transcription.
• whisper-live runs as a Python console app and does something similar.
• Whisper WebGPU runs on the browser (only 200MB). Cool! But slow and still takes up GPU.

uvx --python 3.9 --index https://download.pytorch.org/whl/cu121 whisperx --diarize --lang en --hf_token $HUGGINGFACE_TOKEN

• Idle call time is counted towards cost. So cost may be higher than OpenAI.
• Voice cloning quality is average. Very distinctive voices are just partly identifiable.
• Supports tool calls (from their server).
• Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence.

• Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible].
• Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed.
• Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact.

• Open AI TTS leads the TTS Leaderboard. ElevenLabs is a bit behind.
• Recraft V3 > Flux 1.1 leads Text to Image Leaderboard

• OpenAI TTS: $15/1M chars Ref
• Deepgram Aura: $15/1M chars Ref
• Azure AI Speech: $15/1M chars Ref
• Google TTS Neural2: $16/1M chars Ref
• AWS Polly Neural TTS: $16/1M chars Ref
• Cartesia Pro: $50/1M chars Ref
• Elevenlabs Scale: $300/1M chars Ref

• But the API can "laugh, whisper, and adhere to tone direction." Ref

• OpenAI TTS: $15/1M chars Ref
• Deepgram Aura: $15/1M chars Ref
• Elevenlabs Scale: $165/1M chars Ref
• Google TTS Neural2: $16/1M chars Ref
• Azure AI Speech: $15/1M chars Ref
• AWS Polly Neural TTS: $16/1M chars Ref

• Improve prompts when teaching prompt engineering.
• Pronounce languages better, identifying which words Tamilians and Malayalis will mis-pronounce.

• ElevenLabs speaks in your voice
• Cutout Pro removes backgrounds and parts of images
• Vocal Remover removes vocals from songs
• CapCut video editor