#voice-cloning | Anand S - Things I Learned

Fri, Jan 3, 2025. Assembly AI offers speech to text with diarization at 12c/hour. Good diarization, average transcription quality. #tts #voice-cloning In comparison, WhisperX (with GPU) was much slower, had slightly poorer diarization, and slightly better transcription.
```
uvx --python 3.9 --index https://download.pytorch.org/whl/cu121 whisperx --diarize --lang en --hf_token $HUGGINGFACE_TOKEN`
```

Fri, Nov 29, 2024. GPT-4o Audio supports tone control via XML tags like <cough>..., <laugh>..., etc. But at ~$15/hr of output, it's too expensive. Ref #tts #voice-cloning

Tue, Nov 26, 2024. Ultravox lets you build voice agents at 5c/min = $3/hr (OpenAI is 6c input, 24c output). Or clone their repo. #voice-cloning
- Idle call time is counted towards cost. So cost may be higher than OpenAI.
- Voice cloning quality is average. Very distinctive voices are just partly identifiable.
- Supports tool calls (from their server).
- Their API is simple but the docs have minor errors (e.g. a trailing comma in the JSON, which leads to an error) reducing confidence.

Fri, Nov 8, 2024. Here is a prompt for audio transcription using Gemini. Ref #voice-cloning
- Transcription: Accurately transcribe the audio clip in the original language. Include all spoken words, fillers, slang, colloquialisms, and any code-switching instances. Pay attention to dialects and regional variations common among immigrant communities. Do your best to capture the speech accurately, and flag any unintelligible portions with [inaudible].
- Translation: Translate the transcription into English. Preserve the original meaning, context, idiomatic expressions, and cultural references. Ensure that nuances and subtleties are accurately conveyed.
- Capture Vocal Nuances: Note vocal cues such as tone, pitch, pacing, emphasis, and emotional expressions that may influence the message. These cues are critical for understanding intent and potential impact.

Fri, Nov 8, 2024. ChatGPT for Windows desktop supports real-time voice and a global shortcut (Alt Space). #chatgpt #voice-cloning

Mon, Nov 4, 2024. Hertz-Dev is an open source realtime voice chat model. But it doesn't fit in Google Colab T4's RAM #github #tts #voice-cloning

Tue, Oct 29, 2024. F5-TTS clones voices with just 15-second samples. #future #tts #voice-cloning

Sun, Oct 27, 2024. Elevenlabs lets you create voices with a prompt. No need to even clone one! #tts #voice-cloning

Wed, Oct 9, 2024. Reverb ASR does diarration as well as transcription. It seems the state of art right now. #tts #voice-cloning

Tue, Oct 8, 2024. Revisiting text to speech models. Nothing much has changed since July 2024. #tts #voice-cloning
- OpenAI TTS: $15/1M chars Ref
- Deepgram Aura: $15/1M chars Ref
- Azure AI Speech: $15/1M chars Ref
- Google TTS Neural2: $16/1M chars Ref
- AWS Polly Neural TTS: $16/1M chars Ref
- Cartesia Pro: $50/1M chars Ref
- Elevenlabs Scale: $300/1M chars Ref

Thu, Oct 3, 2024. Speak is a language learning app based on OpenAI's Realtime API. #future #voice-cloning

Thu, Oct 3, 2024. ChatGPT's advanced mode includes: "...you can use various regional accents and dialects." Ref Source #chatgpt #tts #voice-cloning
- But the API can "laugh, whisper, and adhere to tone direction." Ref

Sat, Sep 21, 2024. Sarvam.ai offers Indic text to speech #tts #voice-cloning

Tue, Sep 17, 2024. Segmind's Hallo lets you animate a face to an audio clip #voice-cloning

Fri, Sep 13, 2024. Hume provides a voice-to-voice model (EVI 2) that handles emotions at 7 cents/minute. #voice-cloning

Thu, Jul 25, 2024. Speech editing in audio files is a thing. Speech Editing Toolkit and Descript #ai-coding-tools #voice-cloning

Mon, Jul 8, 2024. A quick check on the pricing of text to speech models #tts #voice-cloning
- OpenAI TTS: $15/1M chars Ref
- Deepgram Aura: $15/1M chars Ref
- Elevenlabs Scale: $165/1M chars Ref
- Google TTS Neural2: $16/1M chars Ref
- Azure AI Speech: $15/1M chars Ref
- AWS Polly Neural TTS: $16/1M chars Ref

Wed, May 29, 2024. Some audio embedding models: #embeddings #voice-cloning unoti/voice-embeddings, retkowsky/audio_embeddings, pyannote/embedding (for speaker similarity), and more.

Sun, Apr 14, 2024. #WRITE fal.ai "animates" pictures, creating videos. It made one from my talk. I morphed into various somewhat similar people rapidly in a 2-second span. Very promising, and far from good. #future #voice-cloning

Sun, Apr 14, 2024. Lemur from Assembly.ai does real time call transcription and summary #future #voice-cloning

Sat, Mar 30, 2024. Hume.ai offers voice emotion API and emotion-based conversational responses. An empathic AI. #future #huggingface #voice-cloning

Mon, Feb 26, 2024. MetaVoice 1B offers voice cloning on American & British accents with 30s training #voice-cloning

Mon, Feb 26, 2024. AI scams are growing. Deepfakes scammed $34m. But voice fake for kidnapping is scarier. #voice-cloning

Mon, Feb 26, 2024. Buildspace's demo is a great demo of how voice and actions can be used effectively. #voice-cloning

Fri, Feb 16, 2024. SORA is OpenAI's video generation model, and is stunning! #voice-cloning

Mon, Jan 15, 2024. Eleven-labs speech synthesis with voice cloning is at the uncanny valley. With two 5-minute samples, my voice sounds a fair bit like my voice but is very clearly not my voice. I find stability ~ 30%, similarity ~ 80% and style ~50% gives a reasonable outcome. But the default voices (e.g. Joseph, George, Charlie) are excellent. #voice-cloning

Sun, Dec 3, 2023. Tools to explore #ai-coding-tools #voice-cloning
- ElevenLabs speaks in your voice
- Cutout Pro removes backgrounds and parts of images
- Vocal Remover removes vocals from songs
- CapCut video editor

Sun, Dec 3, 2023. Meta released SeamlessExpressive which preserves emotions in speech-to-speech translations #embeddings #future #tts #voice-cloning

Sat, Nov 4, 2023. 04 Nov 2023: Seamless4MT does language and speech inter-conversion. CC-NC-BY license #document-conversion #tts #voice-cloning