#embeddings

• Tue, Dec 24, 2024. DuckDB's VSS extension HNSW index + Embeddings (2K chunks of 512 dimensions) takes up roughly 2.5X the size of the original data. Embedding 554 files of ~4,456 KB took 710 seconds. Creating the index took 660 seconds. The resulting DB was 18.1 MB. #embeddings

• Sun, Dec 8, 2024. When embedding using a SentenceTransformer.encode(docs) it's best if we embed with smaller docs and call it multiple times (rather than embedding more at once). On Colab T4, for gte-base-en-v1.5, when embedding 1,000 docs of up to 8K chars each, here is the TOTAL time it took, based on batch sizes (lower is better) #embeddings
  • 1 doc per call: 10s
  • 2 docs per call: 13s
  • 4 docs per call: 19s
  • 8 docs per call: 23s
  • 16 docs per call: 32s
  • 32 docs per call: 40s

• Sun, Dec 8, 2024. Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string. #embeddings #gpu

• Sun, Oct 27, 2024. Via Soumya Ranjan #embeddings
  • Vision embedding is useful in agile modeling
  • Vision embedding models with SAM, Grounding Dino by meta, Alibaba does good stuff
  • Vision embedding is more useful in batch than real time
  • Embedding subtraction with vision embedding models like Dino

• Sun, Oct 27, 2024. Cohere Multimodal Embed v3 is available on Azure. #cloud #embeddings

• Sat, Oct 26, 2024. Meta has a bunch of image embedding models: #embeddings #image-generation
  • DINOv2 creates image embeddings (Apr 2023)
  • ImageBind is an embedding model for text, images, audio, and more (Jun 2023)

• Sun, Oct 6, 2024. Late chunking is an interesting approach to adding context to embeddings. (I don't understand it, but it's cheap and effective.) #embeddings #markdown

• Sun, Oct 6, 2024. DeepInfra offers embedding models as APIs at about 0.5 to 1 cent per MTok in an OpenAI compatible API. #embeddings It also supports text-to-image models like flux.dev and speech recognition models like Whisper.

• Thu, Oct 3, 2024. Matching addresses with just embeddings works well. Combine it with simple hard rules. Source #embeddings

• Sun, Sep 22, 2024. Anthropic's Introducing Contextual Retrieval says: #embeddings
  • Use BM25 in addition to embeddings to match rare terms (e.g. identifiers)
  • Add a context to each chunk's metadata (generate it with a cheap LLM) and pass it to the summarizing LLM
  • Reranking helps with cost AND accuracy. Use Cohere or Voyage

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #TODO #embeddings #future #gpu #markdown #models #optimization #prompt-engineering

• Sat, Jun 8, 2024. Looks like GPT-4o is using CNNs to create vector embeddings of images, with images gridded into a 1x1, 2x2, etc. PLUS OCR. Ref #chatgpt #embeddings #gpu #image-generation

• Wed, May 29, 2024. Some audio embedding models: #embeddings #voice-cloning unoti/voice-embeddings, retkowsky/audio_embeddings, pyannote/embedding (for speaker similarity), and more.

• Wed, Apr 24, 2024. Embeddings can be averaged. So, to embed large documents, average the embeddings of their chunks! OpenAI suggests this. #embeddings

• Sat, Mar 30, 2024. typesense supports embeddings natively. #embeddings

• Sat, Mar 30, 2024. Binary embeddings are good enough. Cohere releases binary embeddings. #ai-coding-tools #embeddings #gpu

• Tue, Feb 13, 2024. Embeddings in random forest are very effective at classification -- much better than dot product. #embeddings

• Sun, Jan 28, 2024. OpenAI releases text-embedding-3-large which can be truncated. The embedding values have descending importance, so picking the first n is a good approximation. Also, gpt-3.5-turbo-0125 is 50% cheaper. #embeddings #gpu

• Fri, Dec 22, 2023. Name2Vec is a potential embedding for names. #embeddings #future

• Sun, Dec 3, 2023. Meta released SeamlessExpressive which preserves emotions in speech-to-speech translations #embeddings #future #tts #voice-cloning

• Sat, Nov 25, 2023. Flat indexing of chunks is not the only way to store embeddings. LlamaIndex allows you to create hierarchies that you can traverse for retrieval #embeddings

• Thu, Nov 23, 2023. Embeddings can help just re-rank regular search results. Ref #embeddings #search