#llm-ops

• Fri, Jan 3, 2025. Does the cost of 'running' a paper/article in an LLM vary depending on the specific LLM used, such as Claude Sonnet? #FAQ #llm-ops
  • Yes, the cost varies depending on the LLM. You can see costs and quality at https://llmpricing.straive.app/. The cost is measured in millions of tokens. For example, the Wikipedia page on the Bible is 100K tokens. You can paste text into https://platform.openai.com/tokenizer to count the number of tokens.Claude 3.5 Sonnet costs $3.5 / MTok, i.e. 35 cents for the Wikipedia Bible page. Gemini 1.5 Flash 8b costs $0.0375 / MTok, i.e. 0.375 cents for the same page. As you can see, the cost can vary by a factor of 100.

• Wed, Jan 1, 2025. LLM Code Execution Sandboxes that let you run code in a sandbox via an API: #cloud #code-agents #github #hosting #llm-ops
  • AgentRun: open source, via Docker
  • e2b.dev. A day costs about $1 (on demand) and you get about $100 one time credits. Self-hosting is complex. Discussion

• Wed, Jan 1, 2025. LLM Observability tools: #cloud #llm-ops
  • LangFuse lets you build a self hosted LLM Proxy with API key distribution, logging, and SSO.
  • Helicone does not support SSO

• Sat, Dec 28, 2024. A clever idea. Give an LLM a chapter from a textbook. Ask it to generate a unique, playable game to help me learn theconcepts for an exam. Page Bailey #llm-ops

• Tue, Dec 24, 2024. How to use LLMs in market research. #llm-ops
  • Use LLMs with search for secondary research.
  • Create different personas and run user surveys on them. This paper used 1,052 real-life interview audio transcripts as agent memory to simulate people
  • Generate your market research report using LLMs.

• Mon, Dec 23, 2024. Medical coding is an area ripe for LLMs. #ai-coding-tools #code-agents #llm-ops
  • Ojasvi Yadav created a repo that uses hierarchical classification (rather than embeddings) to find the right coding.
  • Gemini models seem to understand medical terms better than others.
  • RapidClaims, funded by TogetherAI, is apparently working on this problem.

• Mon, Dec 23, 2024. Awesome Tabular LLMs compiles encodings of tables for LLMs. #llm-ops

• Mon, Dec 23, 2024. What's the best way of encoding tabular data for LLMs? Looks like including the cell address helps. Here is an explanation from ChatGPT #llm-ops

• Mon, Dec 23, 2024. Given about 30 generations, Llama 1b outperforms Llama 8b. Ref #llm-ops

• Sun, Dec 22, 2024. How are software engineers are future-proofing their careers in the face of LLMs? #llm-ops #prompt-engineering
  • Leveraging LLMs as Force Multipliers
    • Use LLMs for repetitive tasks, rapid prototyping, exploring multiple approaches, data extraction and brainstorming, providing feedback.
    • Explore prompting techniques, integrate LLMs into their workflows, and develop strategies for validating and refining LLM-generated code
  • Focusing on higher-level skills that llms struggle with
    • Systems Thinking and Architecture: code readability, extensibility, testability, and maintainability
    • Problem Solving and Critical Thinking: define problems clearly, break them down into manageable parts, and reason through complex scenarios. LLMs produce plausibly incorrect code.
    • Communication and Collaboration
    • Domain Expertise
  • Exploring Adjacent Roles: product management, technical leadership, or consulting. Involve more interaction with clients and stakeholders.
  • Developing "Evergreen" Skills: debugging, system administration, and security. Or outside of software engineering, such as trades or other hands-on vocations.
  • Scepticism: LLMs may not reach a level of sophistication that would render their expertise obsolete. Complex problems, understanding context, and producing high-quality, maintainable code.

• Sun, Dec 22, 2024. Examples of agentic AI #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
  • Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
  • Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
  • Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
  • Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
  • Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
  • Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
  • Meeting notes and action items generator: A system that generates meeting notes and action items.
  • O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
  • UI enhancement agent: An agent that added features like language locales and dark mode to a UI.

• Sat, Dec 21, 2024. Anthropic defines agents. Building effective agents + Cookbook #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
  • Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
    • Prompt chaining: Pipe each LLM output to the next LLM. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
    • Routing: One LLMs decides which other LLM to call next. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
    • Parallelize: Sectioning: Break tasks into independent subtasks, then aggregate. E.g. Evaluate contracts against different clauses in parallel.
    • Parallelize: Voting: Run same task multiple times, then vote. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
    • Orchestrator-workers: One LLM delegates to others, then syntheisizes. E.g. Search from multiple sources, then synthesize. Change multiple code files.
    • Evaluator-optimizer: One model checks another in a loop. E.g. Literary translation. Self-healing code. Policy violation checks.
    • Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages.
  • Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.

• Wed, Dec 18, 2024. SpreadsheetLLM is a way of encoding spreadsheets in an LLM friendly format. #llm-ops

• Thu, Dec 12, 2024. I surveyed the Gramener QA team on how they were using LLMs. #llm-ops
  • 7 used it for code generation (e.g. date extraction, regex generation)
  • 4 used it for learning (e.g. Robot Framework, how to define test cases, API usage)
  • 3 used it for formula generation (e.g. Excel)
  • 2 used it for test scenario identification
  • 2 used it for test data generation
  • 2 used it for comparing expected vs actual datasets
  • 1 used it for data type identification (e.g. given sample values, identify the data type).
  • 1 used it for evaluating resulting (LLM as a judge)

• Thu, Dec 12, 2024. I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: #automation #best-practices #llm-ops #optimization #prompt-engineering
  • Ask better questions. (Prompt engineering.)
  • Create templates or step-by-step instructions. (Chain of Thought.)
  • Ask for multiple options and pick from the best options. (Agentic approach?)
  • Training. (Fine tuning.)
  • Price weaker responses lower. (Stratified model pricing?)

• Thu, Dec 12, 2024. "LLM hallucinations are a good thing. They are a sign of diversity, allowing us to improve the answer by exploring multiple paths." -- A colleague from Straive. #future #llm-ops

• Wed, Dec 11, 2024. Bedrock Llama models can't be directly called with their model names. You need to use their inference profile names, e.g. us.meta.llama3-2-11b-instruct-v1:0 if the model is in a US region. #llm-ops

• Fri, Dec 6, 2024. Arena Hard is a set of hard prompts to test LLMs. Here is the code and evaluation #ai-coding-tools #llm-ops

• Fri, Dec 6, 2024. LLMs can detect clear outliers easily. PROMPT: Which is the outlier in this dataset: (1,7), (2,7), (3,6), (4,6), (5,5), (6,1), (7,5), (8,3), (9,1), (10,1) (ANS: (6,1)) #gpu #llm-ops
  • 🟢 GPT-4o on ChatGPT gets this. GPT-4o Mini on the API gets it too.
  • 🟢 Gemini Pro, Flash, Flash 8b gets this right straight away, without even thinking.
  • 🟢 Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku get it on LLM Foundry. 🔴 Claude.ai, where it visualizes it and gets it wrong.
  • 🟢 Nova Micro, Lite, and Pro get it right.
  • 🟢 Llama 3.1 70b gets it right. 🔴 Llama 3.2 8b gets it wrong. Llama 3.2 70b, Llama 3.1 8b enter repetition.

• Fri, Dec 6, 2024. With LLMs, most architectural decisions are no longer one-way doors. Steve Yegge #future #llm-ops

• Wed, Dec 4, 2024. LLMs are great at convincing you of wrong things. A danger and something to be wary of. Ethan Mollick #llm-ops

• Wed, Dec 4, 2024. Take 100% of your work, including complex, multi step processes and put it into an LLM. It might fail at some but you will discover the limitations. #future #llm-ops

• Wed, Dec 4, 2024. I emailed Straive employees about their use of LLM Foundry - the internal LLM portal. I picked ~500 non-users from teams that otherwise have high (30%+) usage. #llm-ops
  • Reasons they didn't use it were:
    • 40% had not heard of it.
    • 40% were unclear of the benefits
    • 20% didn't have time
  • 45% feel they don't have enough information and training to use it
  • Some feedback
    • Sharing training videos will help
    • Live training sessions that allows for Q&A will help
    • Developers prefer detailed documentation
    • The same prompt gives different results
  • Possible solution: Email non-users introducing the tool and sharing a quick 15-minute tutorial and a 1-page quick start.

• Fri, Nov 29, 2024. Mridula's son gave a live commentary of what he was doing on Minecraft and ChatGPT gave him live evaluation and coaching. E.g. “Great strategy! Getting to the launch pad early can give you a huge mobility advantage. Making the bridge wider is also a smart move to prevent accidental falls. With this plan, you’re setting yourself up for success.” This is a great way to interact with LLMs. #llm-ops

• Wed, Nov 27, 2024. OpenAI allows multiple system messages. I learned this browsing through the LIDA prompts. #automation #future #llm-ops

• Wed, Nov 27, 2024. Anthropic's Model Context Protocol lets any apps integrate with LLM Apps. LLM Apps are becoming the new operating system. Competitors, beware. #future #llm-ops

• Wed, Nov 27, 2024. I spoke at Automating Data Visualizations using LLMs at SUTD. Apparently, using LLMs to write code is much more common than writing code to use LLMs. I ran a quick quiz. #ai-coding-tools #llm-ops
  • Have you used ChatGPT or any LLM? 35 / 35 raised their hands.
  • Have you written code using an LLM? 34 / 35 raised their hands. (I was impressed.)
  • Have you uploaded a spreadsheet to an LLM for analysis? 15 / 35 raised their hands.
  • Have you programmatically called an LLM API? 6 / 35 raised their hands.

• Wed, Nov 27, 2024. With LLMs, fostering innovation is a new path to profitability. Companies are increasing innovation team sizes. Productionizing that is the next. Some initiatives are: #best-practices #future #llm-ops
  • Convert popular demos into starter kits
  • Create and evangelize trainings on solutions and solution techniques
  • Create larger pools of capacity to build innovation and productionize it

• Tue, Nov 26, 2024. LLMs may be good at derived data generation. For example, given a database schema, what derived columns would be useful? What derived views would be useful? #llm-ops

• Mon, Nov 25, 2024. Crawl4AI and Firecrawl are tools / libraries to convert websites into LLM Friendly Markdown and extract structured data using LLMs. #ai-coding-tools #code-agents #document-conversion #html #llm-ops

• Mon, Nov 25, 2024. Don't try and solve specific problems. Pass the entire context to an LLM and get a comprehensive solution. Most doctors, for example, ask specific search-like questions instead of uploading the entire case history and asking for a diagnosis, and perform workse than LLMs. Ethan Mollick #llm-ops

• Fri, Nov 22, 2024. DuckDB has an LLMs.txt. #github #llm-ops #markdown Today, 38 repos on GitHub support it

• Fri, Nov 22, 2024. When identifying LLM use cases, it helps to tell LLMs what they can do. I use one or more of a list like below: #llm-ops
  • Core capabilities:
    • Text Generation: Produce coherent and contextually relevant text across various domains.
    • Image Generation: Create realistic images that match the style and content of a given reference image.
    • Text to Speech: Convert text into natural-sounding speech with appropriate intonation and rhythm.
    • Speech to Text: Transcribe and interpret spoken language.
    • Vision: Analyze and describe visual content from images.
    • Video Analysis: Summarize and extract information from video content.
    • Text to Video: Generate realistic (and surrealistic) videos from text descriptions.
    • Function Calling: Execute predefined functions or access external tools to perform specific tasks.
    • Structured Output: Generate structured outputs like JSON, XML, HTML, YAML, DSLs, etc.
    • Tool Use: Utilize external applications or APIs to enhance functionality.
    • Code Generation: Write and debug code snippets in various programming languages.
  • Cross-domain use cases:
    • Summarization: Understand and condense lengthy documents into concise summaries.
    • Translation: Convert text between multiple languages with high accuracy.
    • Question Answering: Provide precise answers to user queries based on provided information.
    • Reasoning and Planning: Solve complex problems and develop step-by-step plans.
    • Personalization: Tailor responses based on user preferences and historical interactions.
    • Dialogue Management: Engage in context-aware, multi-turn conversations.
    • Data Analysis: Interpret and generate insights from structured data.
    • Content Moderation: Identify and filter inappropriate or harmful content.
    • Sentiment Analysis: Detect and interpret emotions and opinions in text.
    • Robotics Integration: Interface with robotic systems for control and decision-making.
    • Knowledge Retrieval: Access and present information from vast datasets or knowledge bases.
    • Creative Writing: Generate poetry, stories, and other creative content.
    • Educational Assistance: Provide explanations and tutoring across various subjects.
    • Ethical Reasoning: Assess scenarios for ethical considerations and implications.
    • Accessibility Support: Assist users with disabilities through tailored interactions.
    • Simulation and Modeling: Create predictive models and simulate scenarios.
  • Domain-specific use cases:
    • Legal and Medical Assistance: Offer information and guidance within legal and medical domains.
    • Gaming: Generate narratives, dialogues, and scenarios for interactive entertainment.
    • Scientific Research: Aid in literature reviews, hypothesis generation, and data interpretation.
    • Financial Analysis: Analyze market trends and provide investment insights.
    • Cultural Competence: Understand and respect diverse cultural contexts in interactions.
    • Security Applications: Detect and respond to potential cybersecurity threats.
    • Environmental Monitoring: Analyze data related to environmental changes and sustainability.
    • Healthcare Support: Assist in patient monitoring, diagnostics, and personalized treatment plans.
    • Supply Chain Optimization: Enhance logistics and inventory management through predictive analysis.
    • Customer Service: Provide automated support and resolve customer inquiries.
    • Market Research: Analyze consumer behavior and market trends for business insights.
    • Content Creation: Generate articles, blogs, and marketing materials.
    • Virtual Assistance: Manage schedules, reminders, and personal tasks.
    • Social Media Management: Craft posts and engage with audiences across platforms.
    • Human Resources: Assist in recruitment, training, and employee engagement strategies.
    • Event Planning: Organize and coordinate events, including logistics and communication.
    • Travel Planning: Provide itineraries, booking assistance, and destination information.
    • Real Estate: Analyze property markets and assist in buying or selling decisions.
    • Agriculture: Monitor crop health and optimize farming practices through data analysis.
    • Energy Management: Optimize energy consumption and monitor renewable energy sources.
    • Transportation: Enhance route planning and traffic management systems.
    • Urban Planning: Assist in designing sustainable and efficient urban infrastructures.
    • Disaster Response: Provide real-time information and coordination during emergencies.
    • Public Policy: Analyze data to inform policy decisions and predict societal impacts.
    • Art and Design: Generate visual art concepts and assist in creative design processes.
    • Music Composition: Create original music pieces and assist in songwriting.
    • Language Learning: Facilitate language acquisition through interactive exercises and feedback.
    • Historical Analysis: Interpret historical data and provide insights into past events.
    • Philanthropy: Identify charitable opportunities and assess the impact of donations.
    • Sports Analytics: Analyze player performance and game strategies.
    • Fashion: Predict trends and assist in clothing design and merchandising.
    • Culinary Arts: Generate recipes and provide cooking guidance.
    • Astronomy: Analyze celestial data and assist in space exploration research.
    • Psychology: Offer insights into human behavior and mental health support.
    • Linguistics: Analyze language patterns and assist in translation studies.
    • Archaeology: Assist in artifact analysis and historical site interpretations.
    • Literature Analysis: Interpret literary works and provide critical analyses.
    • Philosophy: Engage in discussions on ethical dilemmas and existential questions.
    • Mathematics: Solve complex equations and assist in theoretical research.
    • Physics: Model physical phenomena and assist in experimental design.
    • Chemistry: Analyze chemical compounds and predict reactions.
    • Biology: Assist in genetic research and ecological studies.
    • Geology: Analyze geological data and assist in natural resource exploration.
    • Meteorology: Predict weather patterns and analyze climate data.
    • Oceanography: Study marine ecosystems and assist in ocean exploration.
    • Anthropology: Analyze cultural data and assist in ethnographic research.

• Tue, Nov 19, 2024. Langchain suggests multiple levels of agentic behaviour. LLM Call < LLM Chain < LLM Rounter < State Machine < Autonomous Langchain #code-agents #llm-ops

• Sat, Nov 16, 2024. Anthropic has single-plage docs for LLMs. Condensed version and Full version #future #llm-ops

• Tue, Nov 12, 2024. LLMs pay a lot of attention to the first 6 tokens. Ref #llm-ops

• Sat, Nov 9, 2024. LLM Vision Use Cases in manufacturing and earth sciences (via Shivku) #cloud #image-generation #llm-ops
  • Automated geoscience image descriptions Ref
  • Interpret Wind Turbine photos and charts, construction monitoring, equipment maintenance & charts Ref
  • Forecast weather based on cloud photos! Ref
  • Analyze thermal image of solar panels, electroluminescence images for warranty claims, ROI estimates from Google Sunroof rooftop images Ref
  • Corrosion detection in electricity towers, turbines, storage tanks, penstock. Interpret non-destructive test images Ref

• Mon, Nov 4, 2024. Creating a LLM-as-a-Judge That Drives Business Results by Hamel Husain. #best-practices #dev #llm-ops #prompt-engineering
  • Get THE domain expert (or approver) as the tester.
  • Create a dataset that is DIVERSE.
  • Covers EACH combination of:
    • Features
    • Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error
    • Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user
  • Generate data using existing data + synthetic data for each SPECIFIC combination of the above
  • Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include:
    • Nuances: Something a failed response did well or a passed response didn't quite do well
    • Improvements: Suggest how model can improve
  • Build an SPA to make it easy for the domain expert to review

• Thu, Oct 31, 2024. /llms.txt is a proposal to standardize /llms.txt files as a way to share LLM prompts. #llm-ops #markdown
  • Jina AI Meta Prompt is an example
  • Remotion system prompt is an example
  • https://docs.fastht.ml/llms-ctx.txt
  • https://docs.fastht.ml/llms-ctx-full.txt

• Mon, Oct 28, 2024. NotebookLlama is an open source podcast generator like NotebookLM #llm-ops

• Sun, Oct 13, 2024. LLMs have reached the point where they can show a world, like CounterStrike, in near real time. Ref #future #llm-ops

• Wed, Oct 9, 2024. O1 is good at solving problems where the solution is easy to verify and generating options helps get closer to the solution #future #llm-ops #optimization #prompt-engineering

• Mon, Oct 7, 2024. If you want to create a console.llm() function, a browser extension is the best way, because some pages have Content-Security-Policy that block eval, form submission, fetch from other domains, and script execution. #html #llm-ops

• Sun, Oct 6, 2024. Jake Heller: #future #llm-ops
  • "One of the things we learned is (an LLM app) after it passes passes frankly even 100 tests, the odds that it will do, on any random distribution of user inputs, the next 100,000 100% accurately is very high."

• Sun, Oct 6, 2024. OpenAI's O1 is like Daniel Kahneman's System 2 thinking - as against other LLMs' System 1 thinking. #llm-ops

• Fri, Oct 4, 2024. Aider LLM Leaderboards show which LLMs code better. As of now, #ai-coding-tools #github #llm-ops
  • o1-preview > claude-3.5 sonnet on code editing
  • claude-3-opus > claude-3.5-sonnet on code refactoring
  • deepseek-coder-v
  • gpt-4o-mini sucks.

• Mon, Sep 30, 2024. Coders are more likely to edit LLM code. Non-coders don't have that bad habit. #ai-coding-tools #code-agents #dev #github #llm-ops #markdown
  • Vaishnavi and Ranjeet edited code
  • Indal and Koustav didn't

• Mon, Sep 30, 2024. Coders are likely to get more out of an LLM because they know what it can do. But some non-coders will get more out of an LLM because they don't know what it can't do. #llm-ops
  • E.g. Indal trying for a confetti animation, which is hard but do-able

• Sat, Sep 28, 2024. PyMuPDF4LLM can convert PDFs to Markdown. It handles tables, too. #document-conversion #llm-ops #markdown
  • 04 Oct 2024. PDF-Extract-Kit does PDF layout, formula, table, and OCR extraction using various models.
  • 04 Oct 2024. llmsherpa extracts PDF layout, tables, not OCR

• Fri, Sep 27, 2024. When evaluating feasibility of technology with LLMs always ask for multiple options and pick from those. Simon Willison #future #llm-ops

• Fri, Sep 20, 2024. Ollama 0.2 can handle concurrent requests with only a little additional memory. (So can vLLM and DeepSpeed.) #llm-ops

• Fri, Sep 20, 2024. XML tags seem best to structure prompts across LLMs. #ai-coding-tools #code-agents #llm-ops #markdown #prompt-engineering
  • Claude
  • OpenAI
  • Gemini

• Sun, Sep 15, 2024. Prompt engineering tip: Tell LLMs another AI wrote code. Else they will agree with you! #ai-coding-tools #automation #code-agents #future #llm-ops #prompt-engineering

• Wed, Sep 11, 2024. LLM coding #ai-coding-tools #code-agents #future #llm-ops
  • Makes you more ambitious
  • Lets you code without stress. (Just pass it the error and have it fix it. Or find another approach)
  • Is unlimited. You can run dozens of agents in parallel

• Sat, Sep 7, 2024. I tested out a few audio LLMs: #llm-ops
  • Suno is fast, has a better UI, lots of examples
  • Udio is slow, poor UI, creates richer music

• Sat, Sep 7, 2024. Reflection 70b is one of the top models now, and is open source!. It works by making the LLM reflect on its answer inside <reflection>...</reflection> tags. #future #llm-ops

• Sat, Aug 31, 2024. LLMs are so good that they can simulate Doom in real time. gamengen #llm-ops

• Mon, Aug 19, 2024. LLMs write worse code in JSON than Markdown #ai-coding-tools #llm-ops #markdown

• Wed, Aug 7, 2024. Notes on LLM Fine-Tuning #gpu #llm-ops #optimization
  • Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
  • Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
  • LORA adds additional weights without updating the model. It's a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
  • Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
  • QLORA = Quantization + LORA
  • Predibase has open-sourced Lora Adapters in "Lora Land". Existing adapters are pretty good.
    • ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
    • Locust is a HTTP load testing lib in Python
  • Techniques for inference optimization
    • Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
    • Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters

• Wed, Aug 7, 2024. Notes from a 4-hour flight: #ai-coding-tools #llm-ops #prompt-engineering
  • What We’ve Learned From A Year of Building with LLMs
    • Strategy
      • IS IT TOO HARD/EXPENSIVE? Log it. LLMs are getting cheaper and better.
      • WILL OPENAI BUILD IT? If so, wait for it instead of building.
      • HAS A STARTUP BUILT IT? If so, use it instead. It's a generic use case there's no point re-inventing.
      • FOCUSED USE CASES over generic. Build trust by starting small.
      • Tools for LLM Ops (feedback): LangSmith, Log10, LangFuse, W&B Weave, HoneyHive #TRY
      • Human in the Loop is about humans evaluating model outputs. That's different from AI in the loop, human in the center, where AI accelerates human output (like Github Copilot)
    • Operations
      • CHECK EMBEDDINGS DRIFT over time. Users might be input-ing different things than before.
      • LOG AND REVIEW everything.
      • Instructor coaxes structured output from LLM APIs. #TRY
      • IMPLICIT FEEDBACK collection is easy. Just let users edit stuff. #TRY
    • Tactical
      • Try n-shot prompting (n=5-12) before bigger models. #TRY
      • Always structure for output: Markdown, XML/HTML tags.
      • Combine RAG with Keyword search. It reduces user frustration in edge cases.
      • Prefer multiple small prompts to one big prompt. Do X. Then Y. Then Z.
      • Jitter prompts for diversity beyond temperature.
      • LLM-as-judge works better when comparing outputs (not rating 1 output). Keep length similar (LLMs prefer wordiness). Swap order and compare. Allow for ties. Ask for reason FIRST.
  • Hermes: A Text-to-SQL solution at Swiggy
    • "Hermes performed significantly better for charters with well-defined metadata and a relatively smaller number of tables."
    • "We collect feedback on the accuracy of the returned query from stakeholders directly within the Slack bot."
  • How I use AI and "Replacing my right hand with AI"
    • EMBED in every app/workflow. E.g. Auto-fix spellings. Auto-review code. Auto-ask LLM on errors and apply patch! Auto-search for answer, assess, continue.
    • PERSIST. Stick with the LLM to the end. Don't fix it yourself. It's faster. #TRY
      • INTERVENE FAST. If an LLM can't solve it by itself in 2 tries, it needs in-depth help.
    • APP-IFY one-off tasks. Disposable tools. "Write web-app to convert JSON to tab-delimited." "Extract fields as a table." "Diff JSON." #TRY
    • BEST language/frameworks preferred. CUDA in Python. Rust. C. Raspberry Pi. Arduino. Bluetooth. Modern ESM/JS. #TRY
    • TEACH examples. "Here's the LLM Foundry API." "Here's how to use gramex.data."
    • DUMP entire code. Models can handle it. Refactoring to SQLAlchemy 2, Pandas 2. API Documentation. Test case generation. #TRY
    • ASK for features & packages. Docker without root access. GPU access inside docker. Windows CLI-only C++ compiler.
    • TEST CASE writing. #TRY
    • SPEC IN DETAIL. Use these libraries. Write like this: code example.
    • SPEC USAGE in detail.
      • "I will just pipe it into sqlite", or "I will just run ffmpeg -i filename [YOUR OPTIONS].
      • Describe the UI, API input/output, data structure, and internal data structure.
    • HELP on usage. "ffmpeg to get audio.mp3".
  • My benchmark for large language models
    • LLM(text) is a useful function to have in JS and Python too. Useful as a simple pip install llmfoundry
    • Allow images, files in LLM()
  • Current list of #IMPOSSIBLE (or hard) things for LLMs
    • Translate technical documents to Dutch -- because they don't understand the technical terms well
    • Translate large documents (JSON to XML, English to Chinese, Python to Rust, Wrong to right spelling) -- because the output tokens are limited

• Wed, Jul 24, 2024. GPT 4o Mini is almost as good as GPT 4o in the LMSYS leaderboard. Llama 3.1 400B model and Mistral 2 Large are yet to be evaluated. #chatgpt #gpu #llm-ops

• Fri, Jul 19, 2024. GPT For Work has a set of useful spreadsheet LLM functions #chatgpt #llm-ops

• Sun, Jul 14, 2024. #IMPOSSIBLE LLMs cannot provide a bounding box of objects in images. (Maybe Florence 2 can) #llm-ops

• Tue, Jun 25, 2024. LLM Patterns include Evals, RAG, Fine-tuning, Caching, Guardrails, Defensive UX, Collect feedback. Notably: #ai-coding-tools #automation #best-practices #future #llm-ops #optimization #prompt-engineering
  • Defensive UX: Microsoft, Google, and Apple have guidelines for Human-AI interactions
  • Collect feedback: Explicit and implicit

• Tue, Jun 25, 2024. Rouge and Context Precision are metrics to evaluate LLM responses that serve as a starting point -- but not sufficient, usually #llm-ops

• Sat, Jun 22, 2024. LLM DataHub has LLM training datasets, regularly updated #llm-ops

• Tue, May 28, 2024. Perception of fluency and usefulness are NEGATIVELY correlated in LLM! Evaluating Verifiability in Generative Search Engines #llm-ops

• Sun, May 26, 2024. More LLM routers: #llm-ops
  • LiteLLM: Open source, OpenAI compatible, 100+ LLMs
  • RouteLLM: Open source, OpenAI compatible, automatically routes based on cost
  • OpenRouter: OpenAI compatible API, several models
  • Unify: Supports many models
  • Portkey: Supports popular providers
  • Martian: Limited set of models

• Wed, May 22, 2024. RunPod's vLLM endpoint lets you run any HuggingFace LLM with an OpenAI API priced on usage (serverless) not on idle time. "Autoscaling to 0". #llm-ops

• Sun, May 19, 2024. Portkey is an LLM router #future #llm-ops #networking

• Sat, May 18, 2024. Artificial Analysis benchmarks LLM APIs on speed, cost, and quality. #ai-coding-tools #llm-ops

• Sat, Apr 27, 2024. Tough prompt to test: Gr brx vshdn Fdhvdu flskhu? is a quick way to assess LLM capability. Ref #llm-ops

• Thu, Apr 25, 2024. A long list of LLM courses online #llm-ops

• Thu, Apr 18, 2024. Effort engine introduces "effort" as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now. #gpu #llm-ops #optimization #prompt-engineering

• Sun, Apr 14, 2024. llmsherpa extracts PDFs using LLMs. It has errors but it preserves hierarchy, extracts tables well, and retains image coordinates. Via +91 90031 35354 ~Vetrivel PS #document-conversion #llm-ops

• Sun, Apr 14, 2024. www.web.sp.am is a content farm that's getting hit by OpenAI. Highlights how easy it is to create content farms, and therefore "easy" it can be to introduce bias into LLMs. #ai-coding-tools #llm-ops

• Sun, Apr 14, 2024. Outlines generates structured test with LLMs. It uses the ⭐ logit_bias trick to limit choices in output. See get_choice() #llm-ops #markdown

• Sun, Apr 14, 2024. Data Provenance Explorer lists open datasets used to train LLMs. #llm-ops

• Sun, Apr 14, 2024. Due to in-context learning, giving 100s of examples in the prompt can teach LLMs to jailbreak. Ref #future #llm-ops

• Tue, Apr 2, 2024. Video-Llava is a video LLM #future #llm-ops

• Wed, Mar 27, 2024. Representational engineering lets you control LLM output based on preference on the fly. #future #llm-ops #markdown #models #prompt-engineering

• Wed, Mar 13, 2024. Tavily is a search API for LLMs #llm-ops

• Mon, Feb 26, 2024. Llava 1.6 is a substantial improvement over Llava 1.5 and slightly better than CogVLM, Qwen-VL #llm-ops

• Fri, Feb 23, 2024. My view: LLMs are general purpose and more capable than SLMs. They'll win, like CPUs won over special-purpose chips. GPUs will optimize for LLMs and as usage grows, cost will fall. #gpu #llm-ops

• Sun, Feb 18, 2024. Google is using LLM powered bug identification #github #llm-ops

• Fri, Feb 9, 2024. Fine-tuning session by Dan. Notebook #gpu #llm-ops #markdown
  • Example of fine-tuning Mistral. Consumed 28 computes ($2.8)
  • Axlotl is what the top fine-tuned LLMs are trained on
  • Deepspeed provides distributed training
  • Flash attention lets data stay on GPU
  • Sample packing packs samples of different lengths into equal length tensors

• Sun, Jan 28, 2024. AppAgent is an LLM that can navigate mobile / web apps #code-agents #future #llm-ops

• Sun, Jan 21, 2024. llm-guard is an LLM content validation tool. #llm-ops

• Tue, Jan 16, 2024. Lilac is an LLM-based data curation tool. Use it to search by concept (e.g. PII, duplicates, etc.) and then drop/update the results. #ai-coding-tools #llm-ops

• Sat, Jan 13, 2024. To run an org-specific chatbot on your own LLM: (via awesome-chatgpt) #chatgpt #llm-ops
  • opengpts - but it doesn't support auth
  • chatbot-ui - but Supabase is hard to install
  • anse - but it doesn't support auth
  • ChatGPT-Next-Web - but it doesn't support auth

• Sun, Jan 7, 2024. CodeGPT is an alternative to Github Copilot that can use any LLM. #ai-coding-tools #chatgpt #code-agents #future #github #llm-ops

• Wed, Jan 3, 2024. LMQL provides a constraint-based query language for interacting with LLMs. It uses token masking, which is clever. #llm-ops

• Wed, Jan 3, 2024. Teachers are using LLMs to plan lessons, write emails to parents, create tests, adjust reading level of materials, personalize content with tools like MagicSchool, Diffit, Eduaide. #ai-coding-tools #future #llm-ops

• Tue, Jan 2, 2024. WizardLM creates datasets for instruction tuning by cleverly using LLMs to create new prompts. #llm-ops

• Mon, Dec 25, 2023. Mini-GPTs is an interesting approach to shrink LLMs and make them domain specific. It takes existing LLMs and removes neurons not used in a specific domain (e.g. law, medicine, etc.) #gpu #llm-ops

• Sun, Dec 17, 2023. LLM use cases by Benedict Evans #future #llm-ops
  • "Every text box on the internet will get an LLM"
  • "Infinite interns"
  • "Every UNIX function has become a company." "Every ChatGPT suggestion..."

• Sun, Dec 17, 2023. llm360 publishes models along with training datasets. #future #llm-ops #models

• Sun, Dec 17, 2023. Rubeus is a HTTP proxy for multiple LLMs with load-balancing, fallbacks and retries. #llm-ops

• Sun, Dec 17, 2023. GPTRouter is a Python interface for multiple LLMs with fallbacks and retries. #llm-ops

• Sun, Dec 10, 2023. LIDA from Microsoft is an LLM based data visualization tool. #llm-ops

• Thu, Dec 7, 2023. llama-cpp-python works with other GGUF models like Mistral and allows constrained output - JSON, function calling, etc. Ref #llm-ops

• Tue, Dec 5, 2023. Tools to identify personally identifiable information: #ai-coding-tools #llm-ops
  • galactic can use LLMs to detect PII
  • Presidio by Microsoft
  • Sherlock is a generic sematic type matching DL model
  • pii-extractor-llm was trained on Indian names
  • GLiNER is a Lightweight Generalist model for NER

• Sun, Dec 3, 2023. JupyterLab has a %%ai magic to use LLMs within notebooks. Ref #llm-ops

• Sun, Dec 3, 2023. Unsloth offers faster lower-memory LLM QLoRA finetuning #llm-ops #optimization

• Sun, Dec 3, 2023. DeepSeek is an open-source high-quality LLM #llm-ops #search

• Sun, Dec 3, 2023. SkyPilot lets you run LLMs on any cloud provider. #cloud #future #llm-ops

• Sun, Dec 3, 2023. vLLM lets you deploy LLMs with a single command. #llm-ops

• Sun, Dec 3, 2023. llamafile lets you run LLMs locally as a single file executable! #llm-ops

• Sat, Dec 2, 2023. Gwern Branwen says LLMs nudge his "... making heavier use of the languages I don’t know well (Emacs Lisp & Python) since I increasingly trust that an LLM can help me maintain them." #llm-ops

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding-tools #automation #code-agents #future #llm-ops #optimization #prompt-engineering

• Sat, Nov 25, 2023. Learn ML optimization approach - LLMOps. Have an evaluation framework with metrics like weights and biases or tensorboard. Helps figure out where fine tuning helps and where RAG does. Soon, this will become important. #llm-ops #markdown #optimization

• Sun, Nov 19, 2023. Gorilla LLM specializes in APPI calls: Torch Hub, TensorFlow Hub, HuggingFace #gpu #llm-ops

• Sun, Nov 19, 2023. Ollama runs local LLMs #llm-ops

• Wed, Nov 15, 2023. Tamil-Llama #future #llm-ops

• Mon, Nov 6, 2023. Architecture of today's LLMs is similar to the A16Z architecture #llm-ops

• Mon, Nov 6, 2023. vLLM runs HuggingFace transformers models faster. So does DeepSpeed #future #gpu #llm-ops

• Mon, Nov 6, 2023. 06 Nov 2023: It's possible to extract knowledge, ontologies and taxonomies from LLMs via symbolic distillation #llm-ops

• Sat, Nov 4, 2023. 04 Nov 2023: Running LLMs in production lists experiments to run. Validation CAN be automated! #code-agents #llm-ops

• Tue, Oct 24, 2023. 24 Oct 2023: Neo-sophia has decent videos on LLM strategy #future #llm-ops