#gpu

• Sun, Dec 8, 2024. Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string. #embeddings #gpu

• Fri, Dec 6, 2024. LLMs can detect clear outliers easily. PROMPT: Which is the outlier in this dataset: (1,7), (2,7), (3,6), (4,6), (5,5), (6,1), (7,5), (8,3), (9,1), (10,1) (ANS: (6,1)) #gpu #llm-ops
  • 🟢 GPT-4o on ChatGPT gets this. GPT-4o Mini on the API gets it too.
  • 🟢 Gemini Pro, Flash, Flash 8b gets this right straight away, without even thinking.
  • 🟢 Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku get it on LLM Foundry. 🔴 Claude.ai, where it visualizes it and gets it wrong.
  • 🟢 Nova Micro, Lite, and Pro get it right.
  • 🟢 Llama 3.1 70b gets it right. 🔴 Llama 3.2 8b gets it wrong. Llama 3.2 70b, Llama 3.1 8b enter repetition.

• Wed, Nov 27, 2024. I tried LIDA from Microsoft, after almost a year of its release. A few notes: #ai-coding-tools #gpu
  • Just running uvx lida ui --port 8080 --docs works.
  • But I needed to use export TCL_LIBRARY=C:/Users/Anand/AppData/Roaming/uv/python/cpython-3.13.0-windows-x86_64-none/tcl/tcl8.6 to point it to my TCL installation for charts to work. I also chose to export OPENAI_BASE_URL=https://llmfoundry.straive.com/openai/v1
  • I also chose to replace gpt-3.5-turbo-0301 (the default model) with gpt-4o-mini in lida/web/ui/component*
  • It's quite impressive.

• Fri, Nov 8, 2024. GraphRAG is better if data is naturally graph-structured. Else, it's slow and fills up the context window with even vaguely related stuff. Vigneshbabu, AMAT. #gpu #markdown

• Fri, Sep 27, 2024. Google Vertex AI has an OpenAI compatible API but it works only for some models. Anthropic and Gemini are not compatible. #gpu

• Sat, Sep 21, 2024. E2E is a cheap GPU hosting provider for India. About Rs 100/hr for a V100 16GB #gpu #hosting

• Sat, Sep 21, 2024. Jetson NVIDIa is like Raspberry Pi with a GPU! But it's expensive. #gpu

• Sun, Sep 15, 2024. Groq, SembaNova and Cerebras are fast inference models. All appear to be free #gpu

• Mon, Aug 12, 2024. Some interesting multi-modal generation models / tools to explore: #future #gpu #image-generation #models
  • Flux for open-weights image generation
  • Runway Gen 3 for video generation
  • Suno for music generation

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #TODO #embeddings #future #gpu #markdown #models #optimization #prompt-engineering

• Wed, Aug 7, 2024. Notes on LLM Fine-Tuning #gpu #llm-ops #optimization
  • Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
  • Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
  • LORA adds additional weights without updating the model. It's a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
  • Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
  • QLORA = Quantization + LORA
  • Predibase has open-sourced Lora Adapters in "Lora Land". Existing adapters are pretty good.
    • ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
    • Locust is a HTTP load testing lib in Python
  • Techniques for inference optimization
    • Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
    • Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters

• Wed, Jul 24, 2024. GPT 4o Mini is almost as good as GPT 4o in the LMSYS leaderboard. Llama 3.1 400B model and Mistral 2 Large are yet to be evaluated. #chatgpt #gpu #llm-ops

• Wed, Jul 24, 2024. GPT 4o Mini can be fine-tuned! #chatgpt #future #gpu

• Sat, Jun 22, 2024. Luma Labs Dream Machine generated videos. It's free and is of reasonable quality #cloud #future #gpu

• Sat, Jun 8, 2024. Looks like GPT-4o is using CNNs to create vector embeddings of images, with images gridded into a 1x1, 2x2, etc. PLUS OCR. Ref #chatgpt #embeddings #gpu #image-generation

• Sat, Apr 27, 2024. Cheap cloud GPU services thread on Twitter lists: #ai-coding-tools #cloud #gpu
  • Runpod (17)
  • Vast.ai (17)
  • Modal Labs (8)
  • fly.io (4)
  • LightningAI (4)
  • Colab (4)
  • AkashNet (4)
  • Lambda Labs (4)
  • ShadeFormAI (3)
  • Mac Mini (3)
  • Tensor Dock (2)
  • Hetzner (2)
  • BrevDev (2)

• Thu, Apr 18, 2024. Effort engine introduces "effort" as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now. #gpu #llm-ops #optimization #prompt-engineering

• Sat, Mar 30, 2024. Binary embeddings are good enough. Cohere releases binary embeddings. #ai-coding-tools #embeddings #gpu

• Fri, Feb 23, 2024. My view: LLMs are general purpose and more capable than SLMs. They'll win, like CPUs won over special-purpose chips. GPUs will optimize for LLMs and as usage grows, cost will fall. #gpu #llm-ops

• Wed, Feb 14, 2024. Fly.io offers GPU hosting and auto stop when they have nothing to do. #cloud #gpu #hosting

• Fri, Feb 9, 2024. Fine-tuning session by Dan. Notebook #gpu #llm-ops #markdown
  • Example of fine-tuning Mistral. Consumed 28 computes ($2.8)
  • Axlotl is what the top fine-tuned LLMs are trained on
  • Deepspeed provides distributed training
  • Flash attention lets data stay on GPU
  • Sample packing packs samples of different lengths into equal length tensors

• Sun, Jan 28, 2024. OpenAI releases text-embedding-3-large which can be truncated. The embedding values have descending importance, so picking the first n is a good approximation. Also, gpt-3.5-turbo-0125 is 50% cheaper. #embeddings #gpu

• Sun, Jan 28, 2024. Intel developer cloud has a liberal GPU in the free tier. #cloud #gpu

• Sun, Jan 7, 2024. mixtral-offloading cleverly loads only the model layer required at any point, letting you run Mixtral 8x7b on Colab Free and on 16GB GPUs. This notebook runs on Colab Free too. #cloud #gpu #markdown

• Mon, Dec 25, 2023. Mini-GPTs is an interesting approach to shrink LLMs and make them domain specific. It takes existing LLMs and removes neurons not used in a specific domain (e.g. law, medicine, etc.) #gpu #llm-ops

• Sat, Dec 23, 2023. DPO is a simpler alternative to RLHF for fine-tuning. Several HuggingFace models use DPO for training #gpu #markdown

• Thu, Dec 21, 2023. Generic computate-intensive algorithms eventually beat domain-specific tuning, because of Moore's law. Ref #gpu #optimization

• Wed, Dec 20, 2023. ⭐ This leaderboard included paid models like GPT4 and Claude and compared them with open models on HUMAN + system benchmarks #chatgpt #gpu #models

• Sun, Dec 17, 2023. ⭐ Token Tally has an LLM Cost Tool that estimates GPU memory required and token cost across cloud providers. #gpu

• Sat, Dec 16, 2023. - Grab. Improving last mile delivery in maps. When did people pick up the phone, when should driver be allocated to minimize waiting time, layer on top of OSM. #future #gpu
  • Singapore developers the Sea Lion 7b model
  • Try VLLM with AWQ format. Can do batch inferencing. Needs a good GPU
  • Amex prediction whether they can pay back in 1 year or 18 months. That choice is a business decision. In real time. Precompute individual score and use it as input to another model. Model must be explainable by regulation. Creates decision tree models therefore. Compliance team must agree if I can use a feature. Can't use gender. Age (in US, Canada);- high age is more risk. Can't use edu level in the US.
  • Capture information from camera and use LLMs. Like traffic cameras mapping. Explore GIS from video cameras
  • Grab tracks road closures and road accidents and whether a cycle can go on a road vs a bike vs a car
  • All drivers have a front facing camera
  • Drivers report road accidents by pressing a button
  • Amex prices individual loans when selling to a collection agency
  • #TODO buy a bike head camera!
  • Playwright is a browser-based test framework. Supports recording.

• Sun, Dec 10, 2023. ChatGPT is good at generating questions or training datasets. It genuinely creates them rather than replicating from memory. Ref #chatgpt #gpu

• Sat, Nov 25, 2023. Quote from Jerry Liu: "GPT 4 is really good at complex reasoning". It's worth exploring what that means. #chatgpt #future #gpu

• Sat, Nov 25, 2023. Quote from Jerry Liu: "RAG is a hack". It's engineered, not machine learnt, so it's suboptimal. We need an ML way of creating the context. Maybe fine tuning can be a way of CREATING the right context. But RAG can handle deterministic stuff like access control. #ai-coding-tools #code-agents #gpu

• Sat, Nov 25, 2023. Open AI fine tuning API is not good at memorizing info the way it is exposed. But the Gorilla paper shows that fine tuning can actually memorize well. #gpu

• Sun, Nov 19, 2023. Gorilla LLM specializes in APPI calls: Torch Hub, TensorFlow Hub, HuggingFace #gpu #llm-ops

• Mon, Nov 6, 2023. vLLM runs HuggingFace transformers models faster. So does DeepSpeed #future #gpu #llm-ops

• Mon, Oct 23, 2023. 23 Oct 2023: Bits & Bytes, QLORA, llama.cpp, BigDL, are ways of quantizing models that you can run directly in Google Colab. Try them. Also try OpenVino. #ai-coding-tools #github #gpu #markdown

• Mon, Oct 23, 2023. 23 Oct 2023: Optimum is a set of performance optimization libraries for transformers #gpu #optimization