#voice-cloning

• On the Vibe Coding Process & Strategy
  • The 80/20 Rule is Real: The first 80% of a project is incredibly fast, but the final 20% (debugging, custom features, production-readiness) is extremely difficult and time-consuming.
  • Validation is the New Bottleneck: Since coding is now much faster, the critical, time-consuming task has shifted to reviewing, testing, and validating the LLM's output.
  • "Spec-Locking" is Crucial: Providing the LLM with detailed, well-defined, and "thinly sliced" specifications is essential for getting good results. Vague requests lead to poor outcomes.
  • It's Not Production-Ready (Yet): The consensus is that vibe coding is excellent for prototypes, demos, and go-to-market (GTM) activities but is not yet reliable for building production-grade applications from scratch.
  • Code is Brittle & Unstable: An application that works perfectly one day can inexplicably break the next, as the underlying agent might make undocumented changes.
• Impact on Roles & The Future of Work
  • The Rise of QC/Validation: The Quality Control (QC) function will become larger and more critical to manage the new challenge of validating AI-generated work.
  • Product Managers Shift Focus: PMs can move away from tedious documentation (like flowcharts) and focus more on high-level business strategy, using vibe coding to create quick prototypes.
  • Democratization of Building: It empowers non-coders to build functional apps and helps professionals upskill faster by "conversing" with an LLM on complex topics.
  • New Forms of Cheating: The technology is creating novel ways for people to cheat in interviews, such as using tools that provide real-time subtitles of answers.
  • The "Jagged Edge" of AI: The technology excels at certain tasks (like GTM content) but fails at others, creating new upstream bottlenecks where teams must rapidly generate more of the "AI-friendly" work.
• Practical Hacks & Takeaways
  • Meta-Prompting: Use an LLM to refine and improve your prompt before giving it to the final tool. This helps fill in gaps and add necessary detail.
  • Human-First Drafting: For creative or nuanced work (like writing), it's often better to write the first draft yourself and use the LLM to polish it, rather than starting with a generic AI draft.
  • Use Structured Prompts: For predictable and clean output, providing instructions in a structured format (JSON is OK but not needed) is highly effective.
  • LLM as a Judge: Use LLMs to evaluate and grade content, code, and other outputs, dramatically speeding up the review process.
  • Automate Learning & Documentation: Use tools to transcribe conversations automatically and create personalized revision quizzes from notes and documents.
  • Voice is a Powerful Modality: Using voice-to-code allows for capturing more complex ideas faster and can be done while multitasking (e.g., walking), capitalizing on "dead time."

• Synthesia. Market leader; $2.1B valuation; enterprise trusted. Good: Realism, enterprise features, templating. But: Price, usage caps, slower avatar setup
• HeyGen. Rapidly growing; $500M valuation. Good: Avatar realism, speed, affordability. But: Basic collaboration, support, scene complexity
• Colossyan. Favored L&D focus. Good: Interactive & educational tools, good value. But: Less polished avatars, slower renders
• D-ID. Frequently cited alternative. Good: Speed, flexibility, custom avatars. But: Watermarks, fewer templates
• Elai.io. Repeats in alternatives lists. Good: Storyboarding, educational formats. But: Limited templates, render time
• Hour One. Also common in alternative lists. Good: Photoreal avatars, expression control. But: Missing advanced features like screen capture
• Others. Niche or emerging tools. Good: Varies by platform. But: Less adoption, fewer reviews

• Drift increases faster than linear with hops.
• Bigger models do better, but constrained prompts (“Copy the text exactly; change nothing.”) have a bigger impact.
• Low temperature improves copying fidelity.
• But even after "forgetting", LLMs reproduce rare content if they're trained on it.

• Speech-to-text. Cannot stress this enough but talking to the machine means you’re more likely to share more about what you want it to do.
• I maintain some basic prompts and context for copy-pasting at the end or the beginning of what I entered.
• I ended up preloading executables on the PATH that override the default ones, steering Claude toward the right tools, e.g. running python asks it to use uv.
• I use the task tool frequently for basic parallelization and context isolation.
• Simply taking time to talk to the machine and give clear instructions outperforms elaborate pre-written prompts.
• Forcing myself to evaluate the automation has another benefit: I’m less likely to just blindly assume it helps me.

• Gemini 2.5 Flash Preview TTS ($0.50/1 M input, $10.00/1 M output): ~$0.8 per hour
• GPT-4o-mini-TTS ($0.60/1 M input, $12.00/1 M output): ~$0.9/hour
• Gemini 2.5 Pro Preview TTS ($1.00/1 M input, $20.00/1 M output): ~$1.5 per hour
• GPT-4o-TTS (known as gpt-4o-audio-preview, $2.50/1 M input, $80/1 M output): ~$6.0/hour
• This is comparable to the earlier OpenAI Standard TTS ($0.75), OpenAI HD TTS ($1.5), Google Neural2 ($0.8). ElevenLabs Pro costs ~$6/hr.

• Nomic Embed Multimodal models are the current SOTA on multi-modal embeddings. Notably, they embed PDFs natively.
• Hailuo Speech-02 is the best speech model right now beating ElevenLabs. It has excellent voice cloning. Pricing: $30/1M chars. 10% of ElevenLabs, 2X of OpenAI TTS
• PaperBench is an open testing framework from OpenAI that requires models to replicate the research work in papers. It has ~8,000 tasks evaluated by LLMs and with LLMs judging the judges as well. The code is well worth studying.
• Runway Gen 4 was released with very high character consistency and longer durations
• Dreamina creates lip-synced videos from audio + a single image. Hedra is better for animated characters, though.
• Meta shared but has not released Mocha, an open character generation model that generates new characters speaking based on an audio you provide. It is not based on existing images but the quality is very good
• All Hands has a free online version where you can fix GitHub issues.

• Stickers
  • Sending your wife AI-generated family photos, stickers, etc. is now a thing. Both an AI use case and a ... um... "family media" (?) use case. For example, ask ChatGPT to "Create a transparent comic-style sticker of a lady chef featuring this person happily cooking salad" with a photo. Then send it as a custom sticker. Image
  • Vadivelu stickers work well but the Tamil script generation is poor. Image
• Asking ChatGPT to generate 25-year younger pictures of people produces pretty poor results if you really knew what they looked like then. If you didn't, it's fairly convincing. Yet another example of "hallucinations" - except, it does have its uses.

• Personal writing with connection won't go away. AI can't give you heartbreak. But the rest of non fiction writing will vanish.
• What AI is extraordinary at is personalizing to each audience member's interest
• Outlier opinions will thrive among humans - since AI is trained on consensus.
• Managers tend to be good at working with LLMs because it's mostly about delegation.
• LLMs are perfect for things that don't have a wrong answer! -- Benedict Evans.
• 💡 Explore arguing with AI. It's a safe way to get into a confrontational emotional state (which has its own benefits.)
• 💡 Keep an LLM on in voice mode while reading and ask it any questions you have.
• What models are good for what?
  • GPT 4.5 is great for creation - has a great sense of humor but a corporate style. Still, way better than GPT 4o.
  • ChatGPT is good for voice transcription and note taking. (Increasingly we take notes for AI rather than ourselves.)
  • Claude 3.7 has the best style of writing. It's also great for drawing charts.
  • O1 Pro and Deep Research is great for consumption - research.
  • Grok is the least corporate, able to argue with you, and the latest knowledge cutoff.
  • ElevenLabs for editing podcasts in your voice, making corrections.

• Google's AI overviews now use Gemini 2.0. They've introduced an AI mode that functions like a mini deep research tool, incorporating planning and search. (A Perplexity-killer). It's a fine-tuned model that is extra cautious with topics like healthcare and always verifies information.
• QWQ from Quen competes with DeepSeq R1, but with only 32b parameters compared to R1's several hundred billion.
• AI models are becoming less restrictive. Gemini and GPT-4.5 have relaxed some constraints, shifting more responsibility onto users, similar to Grok.
• What's GPT-4.5 good for? It seems to excel in creativity, humor, education, emotional intelligence, and teaching. It follows instructions better and understands intent better. However, it's not a major leap in coding or math.
• OpenAI's Deep Research mode always uses O3, regardless of the model selected in the UI.
• Tencent has released a new video model available at https://aivideo.hunyuan.tencent.com/ and it appears to be quite good.
• Many clients now support Model Context Protocol (MCP), including Cursor, Claude Code, and Claude Desktop. The clients list is long. Some MCP uses include:
  • Interact with GitHub using the GitHub API.
  • Using Knowledge Graph memory to premember previous conversations
  • Using the Cloudflare MCP server to perform Cloudflare actions.
  • File retrieval and custom prompts -- which MCP supports in addition to tools.
  • Calling other MCPs or LLMs (conditionally) from an MCP, enabling the creation of full-fledged workflows.
• Composio offers a Hosted MCP service. CloudFlare lets you build remote MCP servers.
• Notagen is an open-source note generation engine that produces high-quality classical sheet music.
• Sesame has an open-source voice model worth exploring.
• DiffRhythm is a music generation model that appears to be quite good.

• Deepgram has a MediaRecorder API, which is perfect.
• Whisper Streaming Web is a web app that can transcribe audio real-time from the browser. A good approach, but I wouldn't use it for meeting transcription on my mid-end laptop. Streaming takes up the bulk of my GPU, leaving little for transcription.
• whisper-live runs as a Python console app and does something similar.
• Whisper WebGPU runs on the browser (only 200MB). Cool! But slow and still takes up GPU.

uvx --python 3.9 --index https://download.pytorch.org/whl/cu121 whisperx --diarize --lang en --hf_token $HUGGINGFACE_TOKEN

• Idle call time is counted towards cost. So cost may be higher than OpenAI.
• Voice cloning quality is average. Very distinctive voices are just partly identifiable.
• Supports tool calls (from their server).
• Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence.

• Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible].
• Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed.
• Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact.

• OpenAI TTS: $15/1M chars Ref
• Deepgram Aura: $15/1M chars Ref
• Azure AI Speech: $15/1M chars Ref
• Google TTS Neural2: $16/1M chars Ref
• AWS Polly Neural TTS: $16/1M chars Ref
• Cartesia Pro: $50/1M chars Ref
• Elevenlabs Scale: $300/1M chars Ref

• But the API can "laugh, whisper, and adhere to tone direction." Ref

• OpenAI TTS: $15/1M chars Ref
• Deepgram Aura: $15/1M chars Ref
• Elevenlabs Scale: $165/1M chars Ref
• Google TTS Neural2: $16/1M chars Ref
• Azure AI Speech: $15/1M chars Ref
• AWS Polly Neural TTS: $16/1M chars Ref

• ElevenLabs speaks in your voice
• Cutout Pro removes backgrounds and parts of images
• Vocal Remover removes vocals from songs
• CapCut video editor