#gpu

• Fri, Nov 21, 2025. OpenRouter supports embedding models. BGE base seems pareto optimal with 0.5 cents / MTok and a good MTEB ranking. #embeddings #gpu #models

• Thu, Nov 20, 2025. TOON vs JSON. Early days, and TOON seems to be marketing a lot, so I'm wary, but for large tabular data where input tokens are crunched, it seems a readable alternative to multiple CSVs, but not worth the hype. #chatgpt #gpu #markdown #tts 0 19 Nov 2025. Always use GPT-5.1-Codex-Max instead of GPT-5.1-Codex. At every thinking level, it takes fewer tokens for similar or higher accuracy. Tibo

• Mon, Nov 10, 2025. VaultGemma shows a fine-tuning approach that eliminates personal info that appears only once from memorization. It works by adding noise to weights and capping weights updates so that no one example has undue influence. Model quality is mostly the same. #gpu

• Sat, Nov 8, 2025. Kimi K2 Thinking seems popular because #future #gpu
  • It's an open-weights model on par with the top models on Humanity’s Last Exam (text-only) and BrowseComp
  • Can run 200-300 tool calls without human guidance
  • 4x cheaper than GPT-5 with low tokens (32B active on 1T parameters, INT4 quantized)

• Wed, Nov 5, 2025. Based on responses to Simon Willison's question, ChatGPT #automation #best-practices #future #gpu #optimization #prompt-engineering
  • Fine-tuning helps when:
    • Lower latency, e.g. for type-ahead, at lower cost (37 mentions)
    • Structured extraction, parsing and classifiers, e.g. postal address, detecting secrets (18 mentions)
    • Custom vision models, e.g. check containers (12 mentions)
    • Domain-specific code and stacks (niche languages, stack-specific generation, text→SQL) (11 mentions)
    • ... and a long tail.
  • Fine tuning does not help:
    • When A base model plus prompting or RAG does as well or better (15 mentions)
    • When you risk being leapfrogged by a new release (4 mentions)
    • When cost and data do not justify the ROI (3 mentions)

• Mon, Oct 27, 2025. A nice visual "benchmark" of text-to-image and image editing models. Seadream 4, Gemini 2.5 Flash, and Qwen Image Edit lead. This includes examples like straightening te Tower of Pisa - which only Flux.1 and Seadream 4 do well on; or removing only the brown M&Ms - which only Qwen Image Edit manages to. #ai-art #gpu #image-generation #markdown #models

• Fri, Oct 24, 2025. Comparing the GPT 4.1 and 5 models at all different of reasoning, I've switched my default from GPT 4.1 mini to GPT 5 mini (medium). Far smarter for a slightly higher cost. Artificial Analysis #chatgpt #gpu

• Sun, Sep 28, 2025. Dayflow records your screen at 1 fps and uses Gemini to summarise your activity every 15 min. Has low CPU usage. #dev #gpu

• Mon, Sep 22, 2025. ChatGPT's output is too dense for me. I added this to my custom instructions: "Write in simple language. Explain non-obvious terms intuitively." #chatgpt #gpu #speech-to-text

• Sun, Sep 14, 2025. Depending on the underlying chips that a model uses, the floating point multiplications may differ and model quality can vary. So Claude 4 Opus running on Anthropic's GPUs can produce different results from when running on Google's GPUs or Amazon's GPUs. #gpu

• Wed, Sep 10, 2025. LLMs are non deterministic because GPUs add floating point numbers concurrently and FP addition is non associative - order matters. Thinking Machines #gpu #llm-ops #prediction

• Fri, Aug 22, 2025. DSPy auto-optimizes prompts based on input-output pairs or evals. Typical improvements are ~10-20%. My opinion: avoid. It's a good idea, but has too much abstraction that hides the implementation. Worth learning from but not implementing unless you (a) have evals + metrics and (b) you KNOW you need to change models and (c) it's a long-term project where the learning curve is worth it. Claude and ChatGPT #automation #code-agents #gpu #markdown #optimization #prompt-engineering #speech-to-text

• Sat, Aug 16, 2025. Table OCR Evals. Evaluate how well LLMs parse tables using Promptfoo. #gpu #medium
  • Generate a single image with a table containing
    • 20 rows x 20 columns of psuedo-random (seed-generated) numbers between -1 to +1 million, to 2 decimals.
    • Column & row headers with psuedo-random words
  • Send this to various vision models, e.g. gpt-5-mini, claude-4-sonnet, gemini-2.5-flash, etc. with the same prompt to convert the result into a CSV, re-running this 5 times
  • Compare the result against the original. For each value (text or number) in the original:
    • 0 if the value is in the correct cell
    • -1 if the value is present in an incorrect cell
    • -2 if the value is not present anywhere in the CSV
  • Show the results as a sortable table with columns: Model, Average score, Score in run 1, Score in run 2, ... and cell values are the score, colored on a white-red gradient (white=0, red=most negative score)
  • Next, measure the impact of resolution. Resize the same image to resolutions of 50%, 75%, 150%, and 200%.
  • Repeat the analysis with 3 runs each and compute the average score.
  • Show the results as a sortable table with columns: Model, Average score at 50% resolution, 75%, 100%, 150%, 200% ... with the same cell values and colors.

• Mon, Aug 11, 2025. Apple's Embedding Atlas (Demo - slow, needs WebGPU) is an embeddings visualizer, like #embeddings #gpu #markdown Tensorflow Projector or Mantis (Demo).

• Sat, Jun 21, 2025. As of Jun 21, Claude 3.5 Sonnet > Claude 3.7 Sonnet > O3 Mini > Human > Gemini 1.5 Pro lead the Vending Bench. #gpu Gemini 1.5 Pro also leads my System Prompt Override benchmarks. I'm losing faith in the LM Arena. Perhaps the Gemini models aren't improving as much as we think.

• Thu, Jun 19, 2025. Logprobs are back on models in Vertex AI. Ref #gpu #models

• Mon, May 12, 2025. NVIDIA parakeet is a lightweight speech to text model that leads benchmarks. Installing such packages continues to be a nightmare due to PyTorch (despite uv). #gpu #speech-to-text

• Mon, May 12, 2025. ⭐ Model reliability is a huge enabler for performance. As models become more reliable, they can work autonomously for longer and that is another kind of scaling. Vending Bench #automation #future #gpu #models

• Thu, May 8, 2025. YOLO is much better than Gemini at object detection. Use for pro-processing. Ref #gpu

• Sat, May 3, 2025. Nvidia's OpenMathReasoning 1.5b model beats MUCH larger models at math. Their training dataset is a massive 3.2M rows of math problems with DETAILED thinking traces. #gpu

• Sat, Apr 26, 2025. OpenAI's reasoning models are much ahead of other models when multiplying two numbers in their heads. Ref #gpu

• Fri, Apr 25, 2025. Notes from Latent Space ICLR 2025, Singapore #gpu
  • Daniel: Menlo's ReZero. A model that keeps searching till it finds the answer.
    • There are multiple search techniques: Multi-step retreival, Iterative retrieval, Query rewriting. Also, reasoning.
    • The LLM token generation sequence is normally: <think>, <search>, <answer>.
    • Insight: "If we explicitly reward LLMs for retrying after a failed search, they out-perform one-attempt systems." So <think>, <search>, <think>, <search>, <think>, <search>, <answer>.
    • ⭐ Prompt reasoning models, e.g. "Keep searching till you find the best answer."
  • Roger, Nous Research
    • Supervised learning is limited because accuracy is piece-wise linear, i.e. it's broken up. Continuous optimization is meaningless.
    • Reinforcement learning works better because rewards can be discrete. (But it converts things back into differentiable loss functions behind the scenes.)
      • Rewards can be good/bad. Single or multi-step. Whatever.
    • We're in the "Era of experience", i.e. models gain experience from the environment themselves.
    • ⭐ So, we need environments models can learn in. This is the next thing after training data. That needs a standard for environments.
    • We'd need a model, a trainer, and the environment.
    • The environments whatever capabilities. Run code. Browser. A game. ... With an exposed interface
  • Eugene Cheah (Featherless.ai)
    • Transformer architectures need n-square GPUs as # of tokens grow. Featherless is exploring an RWKV architecture that scales linearly. THere are other such architectures. Performer, Linformer, Reformer, Hyena.
    • Mistral-Nemo-12b-ic is one of the most popular fine-tuned model. It's small enough to run on a server.
  • Justus Mattern (Prime Intellect)
    • Intellect-2 is a continously learning (RL) model that uses decentralized training on peer-to-peer GPUs.
    • Solving problems on bandwidth, verifiable contributions, etc.

• Sun, Apr 20, 2025. The GPT 4.1 models have a 75% discounted prompt caching (instead of the usual 50%), making them particularly suited for repetitive tasks. OpenAI #chatgpt #gpu #prompt-engineering

• Fri, Apr 18, 2025. ⭐ O3 and O4 have built-in tool use covering all of OpenAI's tools, including containers. This allows them to manipulate images and natively understand them improving vision capabilities dramatically. #automation #gpu

• Fri, Apr 18, 2025. GPT 4.1 can handle videos #chatgpt #gpu

• Sun, Mar 30, 2025. LLM Native Multimodal image generation experiments: #chatgpt #gpu #image-generation #voice-cloning
  • Stickers
    • Sending your wife AI-generated family photos, stickers, etc. is now a thing. Both an AI use case and a ... um... "family media" (?) use case. For example, ask ChatGPT to "Create a transparent comic-style sticker of a lady chef featuring this person happily cooking salad" with a photo. Then send it as a custom sticker. Image
    • Vadivelu stickers work well but the Tamil script generation is poor. Image
  • Asking ChatGPT to generate 25-year younger pictures of people produces pretty poor results if you really knew what they looked like then. If you didn't, it's fairly convincing. Yet another example of "hallucinations" - except, it does have its uses.

• Fri, Mar 28, 2025. Notes from ThursdAI - Mar 27 #chatgpt #gpu
  • Gemini 2.5 Pro has good instruction following despite long context. It automatically thinks for longer where required. Good at understanding large codebases. Very fast. You can upload a 2 hour audio to transcribe with timestamps.
  • ai.dev is the shortcut to Google AI studio.
  • ChatGPT native image generation is the best image generation model now. - Great character consistency AND prompt adherence thanks to autoregression and not using stable diffusion. - It tends to refuse image generation less than Dall-E. (While Ghibli-style is possible, Calvin and Hobbes strips are blocked.) "We added a refusal which triggers when a user attempts to generate an image in the style of a living artist." Addendum to GPT-4o System Card - A neat personalization implication is that you could put your kids into their favourite cartoon as a cartoon character that looks like them.
  • It's weird that the latest GPT 4o is ahead of GPT 4.5 on LM Arena.
  • The new DeepSeek V3 is about as good as GPT 4.5 and VERY cheap (27c), so is the obvious choice to run on OpenRouter.
  • MCP news:
    • Qwen.ai supports MCP in the UI! (But it's marked as "coming soon" in my case.)
    • Unlike tools, MCP uses servers that can remember the state or context. Tools are stateless.
    • MCP app store like Smithery, MCP.run, Glama, are mushrooming.
    • Awesome MCP Servers is another good starting point.
    • Azure lets you expose agents as MCP servers.
  • ChatGPT now uses semantic VAD. I interrupts less and typically when you have meaningfully complete something. It responds a little slower as a result.

• Sat, Mar 15, 2025. Here's a sample CI/CD pipeline with automated code review. #github #gpu
  • Here is the script that generated it.
  • Note the use of NVIDIA's GPU Docker containers via nvcr.io

• Fri, Mar 14, 2025. GPT 4o Mini is probably a 8b parameter model. Ref #chatgpt #gpu

• Fri, Mar 7, 2025. Mistral released an impressive OCR model. #document-conversion #github #gpu #markdown
  • Marker from DataLab seems comparable but is CC-BY-NC-SA.
  • MinerU convert medical textbooks to Markdown well.
  • Gemini Flash may be more cost effective and better

• Thu, Jan 30, 2025. Control of chips and GPU compute is what will likely be the gameplay to control AI dominance globally. Dario Amodei #gpu

• Thu, Jan 23, 2025. According to Portkey's LLM usage analysis #cloud #gpu
  • Anyscale and Fireworks AI have the lowest error rates (5xx, 429) and rate limits across providers
  • Groq and Anthropic are among the highest, OpenAI is among the lowest, Google is in-between
  • OpenAI has lower error rates and lower latency than Azure
  • They have a ~35% cache hit rate

• Sun, Dec 8, 2024. Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string. #embeddings #gpu

• Fri, Dec 6, 2024. LLMs can detect clear outliers easily. PROMPT: Which is the outlier in this dataset: (1,7), (2,7), (3,6), (4,6), (5,5), (6,1), (7,5), (8,3), (9,1), (10,1) (ANS: (6,1)) #gpu #llm-ops
  • 🟢 GPT-4o on ChatGPT gets this. GPT-4o Mini on the API gets it too.
  • 🟢 Gemini Pro, Flash, Flash 8b gets this right straight away, without even thinking.
  • 🟢 Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku get it on LLM Foundry. 🔴 Claude.ai, where it visualizes it and gets it wrong.
  • 🟢 Nova Micro, Lite, and Pro get it right.
  • 🟢 Llama 3.1 70b gets it right. 🔴 Llama 3.2 8b gets it wrong. Llama 3.2 70b, Llama 3.1 8b enter repetition.

• Wed, Nov 27, 2024. I tried LIDA from Microsoft, after almost a year of its release. A few notes: #gpu
  • Just running uvx lida ui --port 8080 --docs works.
  • But I needed to use export TCL_LIBRARY=C:/Users/Anand/AppData/Roaming/uv/python/cpython-3.13.0-windows-x86_64-none/tcl/tcl8.6 to point it to my TCL installation for charts to work. I also chose to export OPENAI_BASE_URL=https://llmfoundry.straive.com/openai/v1
  • I also chose to replace gpt-3.5-turbo-0301 (the default model) with gpt-4o-mini in lida/web/ui/component*
  • It's quite impressive.

• Fri, Sep 27, 2024. Google Vertex AI has an OpenAI compatible API but it works only for some models. Anthropic and Gemini are not compatible. #gpu

• Sat, Sep 21, 2024. E2E is a cheap GPU hosting provider for India. About Rs 100/hr for a V100 16GB #gpu #hosting

• Sat, Sep 21, 2024. Jetson NVIDIa is like Raspberry Pi with a GPU! But it's expensive. #gpu

• Sun, Sep 15, 2024. Groq, SembaNova and Cerebras are fast inference models. All appear to be free #gpu

• Mon, Aug 12, 2024. Some interesting multi-modal generation models / tools to explore: #future #gpu #image-generation #models
  • Flux for open-weights image generation
  • Runway Gen 3 for video generation
  • Suno for music generation

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #embeddings #future #gpu #markdown #models #optimization #todo

• Wed, Aug 7, 2024. Notes on LLM Fine-Tuning #gpu #llm-ops #optimization
  • Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
  • Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
  • LORA adds additional weights without updating the model. It's a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
  • Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
  • QLORA = Quantization + LORA
  • Predibase has open-sourced Lora Adapters in "Lora Land". Existing adapters are pretty good.
    • ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
    • Locust is a HTTP load testing lib in Python
  • Techniques for inference optimization
    • Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
    • Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters

• Wed, Jul 24, 2024. GPT 4o Mini is almost as good as GPT 4o in the LMSYS leaderboard. Llama 3.1 400B model and Mistral 2 Large are yet to be evaluated. #chatgpt #gpu #llm-ops

• Wed, Jul 24, 2024. GPT 4o Mini can be fine-tuned! #chatgpt #future #gpu

• Sat, Jun 8, 2024. Looks like GPT-4o is using CNNs to create vector embeddings of images, with images gridded into a 1x1, 2x2, etc. PLUS OCR. Ref #chatgpt #embeddings #gpu #image-generation

• Sat, Apr 27, 2024. Cheap cloud GPU services thread on Twitter lists: #cloud #gpu
  • Runpod (17)
  • Vast.ai (17)
  • Modal Labs (8)
  • fly.io (4)
  • LightningAI (4)
  • Colab (4)
  • AkashNet (4)
  • Lambda Labs (4)
  • ShadeFormAI (3)
  • Mac Mini (3)
  • Tensor Dock (2)
  • Hetzner (2)
  • BrevDev (2)

• Thu, Apr 18, 2024. Effort engine introduces "effort" as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now. #gpu #llm-ops #optimization #prompt-engineering

• Sat, Mar 30, 2024. Binary embeddings are good enough. Cohere releases binary embeddings. #embeddings #gpu

• Fri, Feb 23, 2024. My view: LLMs are general purpose and more capable than SLMs. They'll win, like CPUs won over special-purpose chips. GPUs will optimize for LLMs and as usage grows, cost will fall. #gpu #llm-ops

• Wed, Feb 14, 2024. Fly.io offers GPU hosting and auto stop when they have nothing to do. #cloud #gpu #hosting

• Fri, Feb 9, 2024. Fine-tuning session by Dan. Notebook #gpu #llm-ops #markdown
  • Example of fine-tuning Mistral. Consumed 28 computes ($2.8)
  • Axlotl is what the top fine-tuned LLMs are trained on
  • Deepspeed provides distributed training
  • Flash attention lets data stay on GPU
  • Sample packing packs samples of different lengths into equal length tensors

• Sun, Jan 28, 2024. Intel developer cloud has a liberal GPU in the free tier. #cloud #gpu

• Sun, Jan 28, 2024. OpenAI releases text-embedding-3-large which can be truncated. The embedding values have descending importance, so picking the first n is a good approximation. Also, gpt-3.5-turbo-0125 is 50% cheaper. #embeddings #gpu

• Sun, Jan 7, 2024. mixtral-offloading cleverly loads only the model layer required at any point, letting you run Mixtral 8x7b on Colab Free and on 16GB GPUs. This notebook runs on Colab Free too. #cloud #gpu #markdown

• Mon, Dec 25, 2023. Mini-GPTs is an interesting approach to shrink LLMs and make them domain specific. It takes existing LLMs and removes neurons not used in a specific domain (e.g. law, medicine, etc.) #gpu #llm-ops

• Sat, Dec 23, 2023. DPO is a simpler alternative to RLHF for fine-tuning. Several HuggingFace models use DPO for training #gpu #markdown

• Thu, Dec 21, 2023. Generic computate-intensive algorithms eventually beat domain-specific tuning, because of Moore's law. Ref #gpu #optimization

• Wed, Dec 20, 2023. ⭐ This leaderboard included paid models like GPT4 and Claude and compared them with open models on HUMAN + system benchmarks #chatgpt #gpu #models

• Sun, Dec 17, 2023. ⭐ Token Tally has an LLM Cost Tool that estimates GPU memory required and token cost across cloud providers. #gpu

• Sat, Dec 16, 2023. - Grab. Improving last mile delivery in maps. When did people pick up the phone, when should driver be allocated to minimize waiting time, layer on top of OSM. #future #gpu #ask #todo
  • Singapore developers the Sea Lion 7b model
  • Try VLLM with AWQ format. Can do batch inferencing. Needs a good GPU
  • Amex prediction whether they can pay back in 1 year or 18 months. That choice is a business decision. In real time. Precompute individual score and use it as input to another model. Model must be explainable by regulation. Creates decision tree models therefore. Compliance team must agree if I can use a feature. Can't use gender. Age (in US, Canada);- high age is more risk. Can't use edu level in the US.
  • Capture information from camera and use LLMs. Like traffic cameras mapping. Explore GIS from video cameras
  • Grab tracks road closures and road accidents and whether a cycle can go on a road vs a bike vs a car
  • All drivers have a front facing camera
  • Drivers report road accidents by pressing a button
  • Amex prices individual loans when selling to a collection agency
  • buy a bike head camera!
  • Playwright is a browser-based test framework. Supports recording.

• Sun, Dec 10, 2023. ChatGPT is good at generating questions or training datasets. It genuinely creates them rather than replicating from memory. Ref #chatgpt #gpu

• Sat, Nov 25, 2023. Quote from Jerry Liu: "GPT 4 is really good at complex reasoning". It's worth exploring what that means. #chatgpt #future #gpu #prediction

• Sat, Nov 25, 2023. Quote from Jerry Liu: "RAG is a hack". It's engineered, not machine learnt, so it's suboptimal. We need an ML way of creating the context. Maybe fine tuning can be a way of CREATING the right context. But RAG can handle deterministic stuff like access control. #ai-coding #code-agents #gpu

• Sat, Nov 25, 2023. Open AI fine tuning API is not good at memorizing info the way it is exposed. But the Gorilla paper shows that fine tuning can actually memorize well. #gpu

• Sun, Nov 19, 2023. Gorilla LLM specializes in APPI calls: Torch Hub, TensorFlow Hub, HuggingFace #gpu #llm-ops

• Mon, Nov 6, 2023. vLLM runs HuggingFace transformers models faster. So does DeepSpeed #future #gpu #llm-ops

• Mon, Oct 23, 2023. 23 Oct 2023: Bits & Bytes, QLORA, llama.cpp, BigDL, are ways of quantizing models that you can run directly in Google Colab. Try them. Also try OpenVino. #github #gpu #markdown

• Mon, Oct 23, 2023. 23 Oct 2023: Optimum is a set of performance optimization libraries for transformers #gpu #optimization