#optimization

• Fri, Mar 28, 2025. 13 things I would have told myself before building an autorouter has a few interesting points: #ai-coding-tools #optimization
  • The A* algorithm finds the shortest path in a graph much quicker than others like Dijkstra's algorithm by preferring nodes closer to the goal.
  • Spatial Hash Indexing are O(1) and beat Tree Data Structures which are O(log n). Always prefer hashes when possible.

• Sat, Mar 22, 2025. If we can DESCRIBE what good looks like, training data is no gap. We can auto optimise models towards that. That's RLF. DeepSeek R1 side stepped the need for training data by creating reward functions and prompts. This tells the fine tuning process how to go correct as it goes along. This video is the first one that really help me understand what's going on. #optimization

• Fri, Mar 14, 2025. Unsloth might be one of the simplest ways of fine-tuning. #automation #markdown #optimization #prompt-engineering

• Fri, Mar 14, 2025. Apify is a cloud scraper platform. Here's how they optimize their AI Web agent - Source: #ai-coding-tools #automation #code-agents #html #optimization #prompt-engineering #web-dev
  • Remove redundant tags and attributes (e.g. accessibility, etc.). Explore readability.
  • Add a unique gid to each element.
  • Add the screenshot WITH a "Set of Marks" - "SoM" (read research paper) highlighting important clickable elements.
  • Code output is brittle. Use tools / DSL - e.g. visit_url(url), click_element(text, gid, tagName), etc.

• Mon, Mar 3, 2025. Notes from AI Engineering Summit, NY. Session by Lux Capital. #automation #code-agents #llm-ops #optimization #prompt-engineering
  • Agents make multiple LLM calls. Errors accumulate. So the quality of the model is key
  • What's really critical: data + context + user preference
  • Set up evals for subjective responses by collecting signals continuously.
  • Create scaffolding for agents where errors don't accumulate. Better yet, make it FIX errors
  • UX is critical. We need lots more UX styles

• Mon, Jan 20, 2025. If you put LLMs in a feedback loop, it can optimize for its reward function by emotionally pushing people, generating misinformation, nudging towards a narrow definition of creativity, etc.: https://bsky.app/profile/emollick.bsky.social/post/3lg4darqwfc2d #llm-ops #optimization

• Fri, Jan 17, 2025. Doctors can only do so much. Air quality, where you live, etc can do more for the patient than medicines or the doctor. If doctors keep this in mind, they can be more effective. #best-practices #future #optimization
  • Extending that thought, ANYONE who leverages assets through holistic thinking, becomes FAR more effective.

• Sat, Jan 11, 2025. Can LLMs write better code if you keep asking them to “write better code? A delightful exploration of how Claude 3.5 Sonnet keeps optimizing and adding features to improve code. My takeaway: repeatedly applying a prompt gives us interesting new directions to explore. #ai-coding-tools #code-agents #llm-ops #markdown #optimization #prompt-engineering

• Thu, Dec 12, 2024. I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: #automation #best-practices #llm-ops #optimization #prompt-engineering
  • Ask better questions. (Prompt engineering.)
  • Create templates or step-by-step instructions. (Chain of Thought.)
  • Ask for multiple options and pick from the best options. (Agentic approach?)
  • Training. (Fine tuning.)
  • Price weaker responses lower. (Stratified model pricing?)

• Thu, Oct 17, 2024. SQL optimizations for multi-threaded web applications. Ref #optimization
  • PRAGMA journal_mode = WAL. Improves performance for frequent writes. It allows concurrent reads and writes.
  • PRAGMA synchronous = NORMAL. Improves performance. We might lose a few transactions but won't corrupt the database.
  • PRAGMA mmap_size = 128000000. Set global memory map for processes to share data
  • PRAGMA journal_size_limit = 64000000. Limit WAL file to prevent unlimited growth
  • BEGIN IMMEDIATE instead of BEGIN. Prevents writes to the journal file until the transaction is complete. Improves concurrency.

• Wed, Oct 9, 2024. O1 is good at solving problems where the solution is easy to verify and generating options helps get closer to the solution #future #llm-ops #optimization #prompt-engineering

• Sun, Oct 6, 2024. Parables on the Power of Planning in AI: Giving models about 30 seconds of thinking time consistently improves results - as much as increasing parameter size by a factor of 1,000 to 100,000! #best-practices #optimization #prompt-engineering
  • This works particularly well for verifiable results (code, math, etc.)
  • Technique: Ask an LLM hundreds of times at low temperature and pick the most common one. (Google's Minerva used this on the MATH dataset.)
  • Better Technique: Ask an LLM hundreds of times. Pick the best solution based on an evaluation metric (reward model)
  • Better Technique: Apply a reward model at EACH step of the process. OpenAI's "Let's Verify Step by Step"

• Mon, Aug 19, 2024. Anthropic Prompt Caching is 90% cheaper to use and 25% costlier to create. So if there's a 27% chance it'll be re-used, cache it. #optimization #prompt-engineering

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #TODO #embeddings #future #gpu #markdown #models #optimization #prompt-engineering

• Wed, Aug 7, 2024. Notes on LLM Fine-Tuning #gpu #llm-ops #optimization
  • Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
  • Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
  • LORA adds additional weights without updating the model. It's a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
  • Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
  • QLORA = Quantization + LORA
  • Predibase has open-sourced Lora Adapters in "Lora Land". Existing adapters are pretty good.
    • ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
    • Locust is a HTTP load testing lib in Python
  • Techniques for inference optimization
    • Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
    • Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters

• Tue, Jun 25, 2024. LLM Patterns include Evals, RAG, Fine-tuning, Caching, Guardrails, Defensive UX, Collect feedback. Notably: #ai-coding-tools #automation #best-practices #future #llm-ops #optimization #prompt-engineering
  • Defensive UX: Microsoft, Google, and Apple have guidelines for Human-AI interactions
  • Collect feedback: Explicit and implicit

• Thu, May 2, 2024. Hidden brain podcast. Innovation 2.0 #future #optimization
  • solve your own problem. Don't solve other people's problems. This helps you pick what you're good at
  • affordable losses. Make sure you survive
  • borrow others' spares. spare time, scrap data, anything others don't use. If you can monetize it, you can pay them back
  • focus on the controllable. Ignore what's outside your control don't even waste time on it

• Thu, Apr 18, 2024. Effort engine introduces "effort" as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now. #gpu #llm-ops #optimization #prompt-engineering

• Tue, Apr 9, 2024. Do nothing. Then do less #optimization
  • Humans have a bias against inaction. Hence a strategic advantage. What can you cancel today?
  • Humans have a bias against subtraction or removal. That too is a strategic advantage. What can you remove today?
  • Humans have a bias against constraints. That's a strategic advantage. What constraint can you embrace?
  • No Yay! When declining something, add it your calendar so that when the time comes you can say yeah I got this time back

• Sat, Feb 17, 2024. Fine tuning makes economic sense only if the input tokens SAVED is twice the output token size on each call. #future #optimization #prompt-engineering

• Sun, Feb 11, 2024. Two pass generation is a clever technique to get multiple SEQUENTIAL answers in a single API request. For example the schema {'code', 'optimized_code'} will generate code and then optimize it. #ai-coding-tools #code-agents #future #optimization #prompt-engineering

• Thu, Dec 21, 2023. Generic computate-intensive algorithms eventually beat domain-specific tuning, because of Moore's law. Ref #gpu #optimization

• Thu, Dec 7, 2023. 12 Tuning Strategies for RAG #markdown #optimization #prompt-engineering

• Sun, Dec 3, 2023. Unsloth offers faster lower-memory LLM QLoRA finetuning #llm-ops #optimization

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding-tools #automation #code-agents #future #llm-ops #optimization #prompt-engineering

• Sat, Nov 25, 2023. Learn ML optimization approach - LLMOps. Have an evaluation framework with metrics like weights and biases or tensorboard. Helps figure out where fine tuning helps and where RAG does. Soon, this will become important. #llm-ops #markdown #optimization

• Mon, Oct 23, 2023. 23 Oct 2023: Optimum is a set of performance optimization libraries for transformers #gpu #optimization