#llm-ops

• Ask user (for clarifications)
• Internet tools
  • Search
  • Fetch (CORS-piercing)
  • Scraper with XPath/CSS Selectors
  • Access to llms.txt
• LLM APIs
  • Summarizer (condenses chat)
  • Sub-agents
• Coding tools
  • Markdown convertor
  • Code execution (including tests)
  • Browser + DevTools for testing
• Memory / storage
• Tool/MCP directory with search

• LLMs can't run experiments / explorations, like trying out on a new tool or web app in an environment, the way I would.
• LLMs can't move stuff on my machine, e.g. notes from one list to another, when they're only on my laptop, not GitHub.
• LLMs can't capture the past wisdom in my head, e.g. the distilled principles of data visualization that we applied at Gramener.
• LLMs can't prioritize my to-do list based on my preferences and what's important to me.
• LLMs cannot write a blog post in my style of writing.

• Use vibe coding for stuff you don't need to maintain.
• Use vibe coding for stuff you know well enough to review quickly.
• Use vibe coding for independent tasks where you're not fussed which ones fail.
• Vibe coding turns everyone into a team lead. That needs skills: planning, allocation, design, review, feedback, ...
• ⭐ Empathy enables vibe-coding. Empaths allocate work by ability, review regularly, and provide detailed specs and feedback.
• Have LLMs plan and allocate tasks.
  • "Read this repo. Suggest improvements." (Review.) "Add these as issues."
  • "Add the top 3 Sentry log errors as issues."
  • "Find the easiest issue and solve it with a PR."
• Use GitHub issues extensively for planning.
• ⭐ Create a separate GitHub account for your agent! Let it push. Assign it issues. Treat it like an intern.
• Ensure agent passes test cases and run till the do, or report the core difficulty.
• Throw away rubbish code and start again.
• Issues unsolved in 2-3 tries are too hard for agents or are poorly spec-ed.
• The context7 and Sequential Thinking MCPs are useful.

• LLMs do the task; multiple LLMs cross-check.
• Three tiers: Auto-pass (no human), Light review, Full review.
• Each tier has a clear price and SLA.

• Companies-of-one will grow. Sole founder can handle support functions.
• Specialists will generalize. Consultants will code. Marketers will design.
• Wages will compress. Seniors will earn less as juniors can do more.
• Layers will compress. Organizations need fewer hierarchies as 1 person can do more.
• Shadow apps will grow. Anyone can code. Users build apps with prompts, sheets, agents, outside of IT SDLC. Like Excel sheets.
• Governance will grow. Non-experts are acting like experts. Validation is more important.
• Uneconomical apps will thrive. 1:1 tutoring. Continous decision making or A/B testing.
• Leaders will convince better. Persuasion scales.
• Brand (authenticity, trust, skill), Channel (distribution, audience) and Data are primary differentiators.

• On the Vibe Coding Process & Strategy
  • The 80/20 Rule is Real: The first 80% of a project is incredibly fast, but the final 20% (debugging, custom features, production-readiness) is extremely difficult and time-consuming.
  • Validation is the New Bottleneck: Since coding is now much faster, the critical, time-consuming task has shifted to reviewing, testing, and validating the LLM's output.
  • "Spec-Locking" is Crucial: Providing the LLM with detailed, well-defined, and "thinly sliced" specifications is essential for getting good results. Vague requests lead to poor outcomes.
  • It's Not Production-Ready (Yet): The consensus is that vibe coding is excellent for prototypes, demos, and go-to-market (GTM) activities but is not yet reliable for building production-grade applications from scratch.
  • Code is Brittle & Unstable: An application that works perfectly one day can inexplicably break the next, as the underlying agent might make undocumented changes.
• Impact on Roles & The Future of Work
  • The Rise of QC/Validation: The Quality Control (QC) function will become larger and more critical to manage the new challenge of validating AI-generated work.
  • Product Managers Shift Focus: PMs can move away from tedious documentation (like flowcharts) and focus more on high-level business strategy, using vibe coding to create quick prototypes.
  • Democratization of Building: It empowers non-coders to build functional apps and helps professionals upskill faster by "conversing" with an LLM on complex topics.
  • New Forms of Cheating: The technology is creating novel ways for people to cheat in interviews, such as using tools that provide real-time subtitles of answers.
  • The "Jagged Edge" of AI: The technology excels at certain tasks (like GTM content) but fails at others, creating new upstream bottlenecks where teams must rapidly generate more of the "AI-friendly" work.
• Practical Hacks & Takeaways
  • Meta-Prompting: Use an LLM to refine and improve your prompt before giving it to the final tool. This helps fill in gaps and add necessary detail.
  • Human-First Drafting: For creative or nuanced work (like writing), it's often better to write the first draft yourself and use the LLM to polish it, rather than starting with a generic AI draft.
  • Use Structured Prompts: For predictable and clean output, providing instructions in a structured format (JSON is OK but not needed) is highly effective.
  • LLM as a Judge: Use LLMs to evaluate and grade content, code, and other outputs, dramatically speeding up the review process.
  • Automate Learning & Documentation: Use tools to transcribe conversations automatically and create personalized revision quizzes from notes and documents.
  • Voice is a Powerful Modality: Using voice-to-code allows for capturing more complex ideas faster and can be done while multitasking (e.g., walking), capitalizing on "dead time."

• After sending a response, send the prompt + input + output + optional user feedback to an LLM-as-a-judge asking for feedback to improve the prompt.
• Revise the prompt based on the improvement. Now the app has improved, real-time, based on human/LLM feedback.
• Refine this process to ensure that the revisions are smooth and positive.

• 1/2: Become an API to LLMs. Create an API for LLMs to consult you. Answer based on past chats, notes, memory, etc.!
• 2/2: From a prompt, have an LLM build synthetic data, the OpenAPI spec, a CloudFlare web app, deploys it, and update your portfolio -- automatically!

• Drift increases faster than linear with hops.
• Bigger models do better, but constrained prompts (“Copy the text exactly; change nothing.”) have a bigger impact.
• Low temperature improves copying fidelity.
• But even after "forgetting", LLMs reproduce rare content if they're trained on it.

• Use the best models, consciously. O3 (via $20 ChatGPT), Gemini 2.5 Pro (free on Gemini app), or Claude 4 Opus (via $20 Claude). The older models are the default and far worse.
• Speak & listen, don't just type & read. I had to resist the temptation to ignore ChatGPT response when a colleague read it out. We are patient with and have respect for humans but not for AI. The value we derive requires both. Suggestion: Speak and listen rather than type and read. It's hard to skip and easier to stay in the present. It's also easier to ramble than type.
• Keep an impossibility list. There is a jagged edge that moves. When you note down what's impossibile today and retry every month, you can see how that edge shifts.
• Wait for better models. Many problems can be solved just by waiting a few months for a new model. You don't need to find or build your own app.
• Make context easily available. Context is one of the biggest enablers for LLMs. Use search, copy-pasteable files, previous chats, connectors, APIs/tools, or any other way to give LLMs examples and context.
• Have LLMs write code. LLMs are bad at math. They're good at languages, including code. Running the code gives output with low hallucinations. This combination can solve a WIDE variety of problems that need creativity and reliability.
• Learn AI coding. 1. Build a game with ChatGPT/Claude/Gemini. 2. Improve it. 3. Create a tool useful to you. 4. Publish it on GitHub.
• APIs are cheaper than self hosting. Avoid self-hosting.
• Datasets are more important than fine-tuning. You can always fine-tune a newer model as long as you have the datasets.

llm --save ffmpeg --model gpt-4.1-mini --extract --system 'Write an ffmpeg command'

llm -t ffmpeg 'Crossfade a.mkv (1:00-1:30) with b.mkv (2:10-2:20), 3s duration'

• Provide a stable, complete spec.
  • Late UI tweaks, new API params, renamed fields, extra packaging rules, “Rename per‑image download”, “standardise baseUrl vs baseURL”, “add GA‑4 exam module”. → churn & rewrites.
  • Ask the user for a final UI/API/mock‑up + edge‑case examples before the first commit.
  • Lock naming conventions, UI layout and feature checklist early; track future changes explicitly
• Include concrete examples.
  • Lack of sample images, Markdown snippets, question formats caused guesswork.
  • Supply mini‑fixtures: sample prompts, expected outputs, env‑var names, commit‑message template
• Environment should be reproducible.
  • E.g. vitest not installed, .dev.vars absent, sub‑modules not cloned, network blocks.
  • Ship a one‑step bootstrap script / README with npm install, env‑var templates, and submodule notes
• Automate tests.
  • First answer compiles but fails prettier/ruff/unit tests; later iterations fix style or red lines.
  • Codex should auto‑run lint && test (plus static‑analysis / self‑critique) before every response
• Auto-run post-mortems.
  • Codex recommending its own static checks shows value.
  • Automate that as a pre‑commit step.

• Documents. E.g. Docsearch, GPTs, Microsoft Copilot, Gemini
• Slides. E.g. Microsoft Copilot, Gemini
• Sheets. E.g. Microsoft Copilot, Gemini
• Code. E.g. Cursor, Claude Code
• Database. Create DB schema, ER diagrams, synthetic data, ingestion scripts, etc.
• Data (analysis). E.g. Datachat, Google Colab, Marimo
• Posters. E.g. Postgen
• Shell. E.g. Warp
• Topic modeling. E.g. classify
• Surveys. E.g. Personagen
• APIs. E.g. apiagent
• Drug regulatory submissions.
• Contracts (risk).
• Manufacturing SOPs.
• Curriculum.
• Data quality.
• Support tickets.
• Dashboards.
• IaaC / DevOps.
• Video campaigns.
• Resumes.
• Patents.

• Increase clarity, providing step-by-step instructions.
• Resolve conflicting instructions.
• Expand instructions to cover all scenarios and edge cases.

• Using foundation model evals instead of application evals is like evaluating a candidate on SAT scores. It's fine, but you also want to evaluate them on their specific job description.
• Evals must be done by the users and not outsourced.
• Evals are not draining.
• Small samples have high value.
• When using LLM as a judge, be VERY VERY specific about the criteria.
• Prefer binary LLM evals over scales.
• Monitor performance online, not just while deploying

• Sam Altman. Chase $0B ideas, not $0M ones. Weird + right > safe + crowded
• Gary Tan. Agency scales. Tools change, people/mindset don’t.
• Andrej Karpathy.
  • Instead of LLM memory to store facts, edit system prompt with general strategies, like the LLM writing a book for itself on how to solve problems.
  • Autonomy slider. Let user pick how far LLM acts by itself. Like the Tesla autopilot levels.
  • Make evals EASY and FAST for humans.

• Practice Prompt Reviews. Check if each prompt has clarity, context, and verifiability. Also, see how others would ask this. Internalize patterns

• Is RAG dead? Retrieval is not. Naive vector search is less popular. Hybrid > Vector search. Tools work better for code. SQL works better for data.
• Same model for task + evals is OK? Yes. Pick a good model for evals.
• Is model choice critical? Only if evals tell you so.
• Should I build a custom annotation tool? Yes, always. Your data and workflow is unique.
• Why binary evals not Likert scales? For clearer and more consistent labelling.
• How do I debug multi-turn chats? Manually review failures. Reproduce the simplest possible test case. Provide N-1 real chats and test the failure point.
• Should I build automated evaluators? Only for failures that persist after fixing prompts.
• How many human evaluators? Prefer one benevolent dictator. For complex problems, measure evaluator alignment with Cohen’s Kappa.
• What beyond evaluator tool?
  • Cluster errors for patterns.
  • LLMs for EDA on logs and fixes.
  • Build custom evaluators.
  • Integrate with annotator tool APIs.
• How to generate synthetic data? List dimensions & values. Prefer high-failure values. Then create combinations.
• How to evaluate unknown/diverse queries? Do error analysis. Don't pre-determine evals.
• What's the right chunk size? For pointed answers, pick largest relevant chunk. For synthesis (summarize, list), pick smaller chunks.
• How to evaluate RAG? See 6 RAG Evals.
  • Retrieval: Recall@k, Precision@k, MRR
  • Generation: Error analysis, human labeling, LLM-as-judge
• What UI for evals? Align to domain. Show progress. Support keyboard. Allow filter, cluster, search. Prioritize problematic traces. Keep it minimal.

• Write very comprehensive specs.
• Use LLM to create the specs.
• Reviewing is a skill we need to develop.
• Understanding others' code takes effort.
• But LLM code is easier to review because it's immediate and has no ego.
• Unit tests are critical.
• Use LLMs for well understood specs, APIs, platforms and libraries to really save time.
• Logic-less stuff like Markdown, JSON and HTML templates are a LOT easier to verify. Do more of that.
• We can only make so many decisions in a day. AI coding saves us that effort.
• Experts are not experts in every area. They benefit from LLMs in other areas.
• LLMs are great for rubber ducking. Speaking and speccing really help.
• LLMs make mistakes. So do most humans.
• LLM speed makes coding more exhausting.
• Use LLMs to understand codebases.
• AI coding could reduce demand for developers. E.g. Sysadmin demand plummeted with cloud infra and infrastructure-as-code.
• But, niche use cases could grow, like how demand for photographers grew despite point-and-shoot cameras.
• Transaction cost of hiring even 1 person is high and that will likely be a bottleneck. Plus people can use LLMs themselves, so that will dampen niche demand.

• Responses API allows reasoning tokens to be encrypted if organizations don't want their reasoning data to persist. Ref
• Tinfoil (YC X25) offers an OpenAI-compatible inference API where data is encrypted from the client to the NVIDIA Hopper/Blackwell GPUs in confidential computing mode. Prompts, model weights, outputs are encrypted in transit and memory, with verifiable privacy on code running in GPU.
• Modelyo (Israel) offers VMs/K8 clusters with encrypted GPUs across multiple cloud providers with continuous attestation, managed on Modelyo's portal.

• GPT 4o: started giving its system prompt: "You are ChatGPT, a large language model trained by OpenAI. Knowledge cutoff: 2024-06. Current date: 2025-04-27. Image input capabilities: Enabled. Personality: v2. ..."
• O4 Mini: Refused to comply
• Gemini 2.5 Flash: Gave me my custom instructions.

• Recurse center (Brooklyn, online) is a 6/12 week free self-driven programmer retreat. Runs every 6 weeks. You can do whatever you pick. There are daily standups for accountability. The groups are diverse. You can pair with them, pivot ideas, whatever. Principles: push yourself & learn.
• Western education techniques (e.g. spaced repetition, adaptive learning) are very much present in Indian coaching systems, though not known by those names. However, interventions are hard since class 12 students just don't have enough time.
• Coaching classes are a social phenomenon. It's not the smart students who are pulling in their friends. Smart students actually follow the popular students. (Coaching classes are below the typical smart students' standards.)
• Monetizing coaching is hard. People don't want to pay for advice, and welcome free advice only if they ask for it. Coupling with execution is necessary.
• Aider's integrations make it more powerful than Cursor/Windsurf. It auto-lints, runs test cases. Allows different models for "architecting" (generating changes) vs "editing" (applying code). It reads from the screen logs. Context is manual, not automated. Uses an ai! comment to trigger changes and ai? to ask questions.
• Cline.bot is another Cursor-like open source AI code editor that's a VS Code plugin.
• When coding with LLMs, a useful workflow is: data schema ➡️ interfaces ➡️ LLM-generated test cases ➡️ code.
• ShellSage is a tmux based LLM tool for the command line. It screen-grabs from tmux, which is powerful.
• Some MCPs that have proven useful: vega-lite, SQLite, sequential thinking, memory
• make sucks but is hard to beat. just comes closest.
• CRDTs are more powerful than for just collaborative editing. It can power a peer-to-peer Internet (beginning with office tools). Versioning schema is still problematic. yjs is a good start but automerge (Rust, WASM) is faster and may be better.
• Fermyon hosts WASM serverless functions.

• Marketing and Advertising: Ad Copy and Campaign Content Generation, Personalized Marketing Messages, Creative Strategy Brainstorms, Automated Marketing Production (Everyday Wins)
• Customer Service and Support: AI-Powered Chatbots for Common Queries, Agent Assist and Email Drafting, Summarizing and Analyzing Customer Feedback, Interactive Troubleshooting and FAQs
• Retail and eCommerce: AI-generated Summary of Product Reviews, Product Description and Catalog Content Generation, Visual Content and Image Captions, Personalized Shopping Recommendations (Narrative Form)
• Human Resources and Talent Management: Job Description and Policy Writing, Resume Screening and Candidate Q&A, Employee Communications and Feedback, Training and Onboarding Content
• Education and E-Learning: Personalized Explanations and Tutoring, Content Creation: Stories, Examples, and Analogies, Practice Problems and Quiz Generation, Automated Grading and Feedback
• Media and Entertainment: Writing and Editing Assistance, Personalized Media Content, Localization and Dubbing Scripts, Content Moderation and Curation (Assistive)
• Finance and Banking: Market Commentary and Research Summaries, Client Communications and Explanations, Regulatory Compliance Summaries, Scenario Analysis and Planning
• Management Consulting and Strategy: Research and Insight Generation, Document and Slide Drafting, Brainstorming and Scenario Planning
• Legal Services: Drafting Contracts and Legal Documents, Legal Research Q&A and Summaries, Client Communications and Explanations

• The effort for evals may seem high. Use LLMs to reduce this effort.
• Include irrelevant questions because people WILL ask them. Be clear on how to handle that.

• Keep researching. Use LLMs as an altemative to books and other reading material. Keep publishing what you learn regularly.
• While reading a chapter, keep asking the LLM. What did you think of that? What just happened there? What should I focus more on? What's puzzling about this? How do I connect this to something else later or earlier in the book?
• LLM is better used to support you rather than replace you in areas of your expertise. Where you are an expert it's best for you to be yourself and have AI fill in the gaps.
• Ask the AI: "What is in my writing that some people might find obnoxious? Or cold / heartless? Explain it to me in great detail."
• The first input is context setting and should be really long. Use voice dictation for that instead of typing.
• Send your blog post to an LLM. No need to explain it. Just let it be the reader and see what it understands and doesn't understand.
• His PhD students don't have a textbook, which saves them some money. But they are required to subscribe to a large language model which ends up costing less.
• Today, it makes sense to use the best models and pay $200 for it if required. The differences are large. But in some years in the future, the cost of these models may come down for the free versions.
• Humans know secrets. AI does not. So at least in some areas, humans will have an advantage.
• Secrets full matter a lot more in the future. Gossip will matter a lot more. How good are you at keeping and trading secret? Travelling and meeting people will become more important. So will the value of social networks.
• Since everyone has access to better intelligence, the value of mobilization or being able to do things with people will have higher value. Leadership is an example. The value of your network therefore has gone up a lot.
• There's more value in prompting one thing 10 times then 10 things one time. Follow up questions work better than long prompts.
• There are so many AI note-takers (and transcribers) these days that you are not just writing for an AI but speaking for AIs as well!
• Which model to use:
  • O1 Pro is the best model. Claude does a decent job. DeepSeek is full of hallucinations but is interesting. It is more imaginative.
  • Use O3 mini to write your prompt first, and then ask the model
  • Use DeepSeek and other somewhat wacky high-end models once a day so that you stay in touch with what is models are capable of (beyond the conventional.)
  • Perplexity has entirely replaced Google for many people.
  • Anthropic's models are the best writers.
  • Gemini is good for long documents and hence for things like legal work. Gemini also has excellent YouTube integration and hands can directly read the transcripts.
  • Grok is very good at fact checking tweets.
• Converting data into LLM consumable forms will be a huge project. Lot of a knowledge is not in such a form and a huge human project will involve this conversion.

• Model errors. The model itself makes a mistake. E.g. hallucinations, not following the prompt, etc.
• Context errors. The model makes a mistake because the question was out of context, or the context was missing.
• Input errors. The input to the model was parsed incorrectly, e.g. poor audio, poor image OCR, etc.
• Tool errors. The model's tools are wrong or not good enough, e.g. Retrieval errors.

• Agents make multiple LLM calls. Errors accumulate. So the quality of the model is key
• What's really critical: data + context + user preference
• Set up evals for subjective responses by collecting signals continuously.
• Create scaffolding for agents where errors don't accumulate. Better yet, make it FIX errors
• UX is critical. We need lots more UX styles

• LLMs are clearly not machines. They give different answers each time.
• LLMs are like humans: they exhibit human biases (e.g. guessing 42 or 37 often). But they fail in unusual ways. They can't count the "r"s in strawberry. They can go into an endless loop.
• LLMs are a new form of intelligence. Thinking of them as aliens might minimize our confusions.

• Yes, the cost varies depending on the LLM. You can see costs and quality at https://llmpricing.straive.app/. The cost is measured in millions of tokens. For example, the Wikipedia page on the Bible is 100K tokens. You can paste text into https://platform.openai.com/tokenizer to count the number of tokens.Claude 3.5 Sonnet costs $3.5 / MTok, i.e. 35 cents for the Wikipedia Bible page. Gemini 1.5 Flash 8b costs $0.0375 / MTok, i.e. 0.375 cents for the same page. As you can see, the cost can vary by a factor of 100.

• AgentRun: open source, via Docker
• e2b.dev: A day costs about $1 (on demand) and you get about $100 one time credits. Self-hosting is complex. Discussion
• nsjail: by Google. Write your own API

• LiteLLM is an LLM Proxy with caching, logging, call hooks or plugins, rate limiting, virtual keys. SSO integration can be implemented.
• LangFuse is an LLM Proxy with API key distribution, logging, and SSO. But lacks per-user usage limits and server-side caching.
• Helicone does not support SSO

• Use LLMs with search for secondary research.
• Create different personas and run user surveys on them. This paper used 1,052 real-life interview audio transcripts as agent memory to simulate people
• Generate your market research report using LLMs.

• Ojasvi Yadav created a repo that uses hierarchical classification (rather than embeddings) to find the right coding.
• Gemini models seem to understand medical terms better than others.
• RapidClaims, funded by TogetherAI, is apparently working on this problem.

• Leveraging LLMs as Force Multipliers
  • Use LLMs for repetitive tasks, rapid prototyping, exploring multiple approaches, data extraction and brainstorming, providing feedback.
  • Explore prompting techniques, integrate LLMs into their workflows, and develop strategies for validating and refining LLM-generated code
• Focusing on higher-level skills that llms struggle with
  • Systems Thinking and Architecture: code readability, extensibility, testability, and maintainability
  • Problem Solving and Critical Thinking: define problems clearly, break them down into manageable parts, and reason through complex scenarios. LLMs produce plausibly incorrect code.
  • Communication and Collaboration
  • Domain Expertise
• Exploring Adjacent Roles: product management, technical leadership, or consulting. Involve more interaction with clients and stakeholders.
• Developing "Evergreen" Skills: debugging, system administration, and security. Or outside of software engineering, such as trades or other hands-on vocations.
• Scepticism: LLMs may not reach a level of sophistication that would render their expertise obsolete. Complex problems, understanding context, and producing high-quality, maintainable code.

• Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
• Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
• Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
• Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
• Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
• Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
• Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
• Meeting notes and action items generator: A system that generates meeting notes and action items.
• O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
• UI enhancement agent: An agent that added features like language locales and dark mode to a UI.

• Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
• Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
  • Prompt chaining: Pipe each LLM output to the next LLM. A->B->C->Z. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
  • Routing: One LLMs decides which other LLM to call next. A->B|C|D->Z. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
  • Parallelize: Sectioning (and Orchestrator-workers): Break tasks into independent subtasks, then aggregate. A->B+C+D->Z. E.g. Evaluate contracts against different clauses in parallel.
  • Parallelize: Voting: Run same task multiple times, then vote. A->B+B+B->Z. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
  • Evaluator-optimizer: One model checks another in a loop. A->B->A->B->...->Z. E.g. Literary translation. Self-healing code. Policy violation checks.
  • Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages. A->B->(Human)->C->Z. E.g. Sensitive content review. High-stakes decision making. Ambiguous tasks.
• Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.

• 7 used it for code generation (e.g. date extraction, regex generation)
• 4 used it for learning (e.g. Robot Framework, how to define test cases, API usage)
• 3 used it for formula generation (e.g. Excel)
• 2 used it for test scenario identification
• 2 used it for test data generation
• 2 used it for comparing expected vs actual datasets
• 1 used it for data type identification (e.g. given sample values, identify the data type).
• 1 used it for evaluating resulting (LLM as a judge)

• Ask better questions. (Prompt engineering.)
• Create templates or step-by-step instructions. (Chain of Thought.)
• Ask for multiple options and pick from the best options. (Agentic approach?)
• Training. (Fine tuning.)
• Price weaker responses lower. (Stratified model pricing?)

• 🟢 GPT-4o on ChatGPT gets this. GPT-4o Mini on the API gets it too.
• 🟢 Gemini Pro, Flash, Flash 8b gets this right straight away, without even thinking.
• 🟢 Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3.5 Haiku get it on LLM Foundry. 🔴 Claude.ai, where it visualizes it and gets it wrong.
• 🟢 Nova Micro, Lite, and Pro get it right.
• 🟢 Llama 3.1 70b gets it right. 🔴 Llama 3.2 8b gets it wrong. Llama 3.2 70b, Llama 3.1 8b enter repetition.

• Reasons they didn't use it were:
  • 40% had not heard of it.
  • 40% were unclear of the benefits
  • 20% didn't have time
• 45% feel they don't have enough information and training to use it
• Some feedback
  • Sharing training videos will help
  • Live training sessions that allows for Q&A will help
  • Developers prefer detailed documentation
  • The same prompt gives different results
• Possible solution: Email non-users introducing the tool and sharing a quick 15-minute tutorial and a 1-page quick start.

• Have you used ChatGPT or any LLM? 35 / 35 raised their hands.
• Have you written code using an LLM? 34 / 35 raised their hands. (I was impressed.)
• Have you uploaded a spreadsheet to an LLM for analysis? 15 / 35 raised their hands.
• Have you programmatically called an LLM API? 6 / 35 raised their hands.

• Convert popular demos into starter kits
• Create and evangelize trainings on solutions and solution techniques
• Create larger pools of capacity to build innovation and productionize it

• Core capabilities:
  • Text Generation: Produce coherent and contextually relevant text across various domains.
  • Image Generation: Create realistic images that match the style and content of a given reference image.
  • Text to Speech: Convert text into natural-sounding speech with appropriate intonation and rhythm.
  • Speech to Text: Transcribe and interpret spoken language.
  • Vision: Analyze and describe visual content from images.
  • Video Analysis: Summarize and extract information from video content.
  • Text to Video: Generate realistic (and surrealistic) videos from text descriptions.
  • Function Calling: Execute predefined functions or access external tools to perform specific tasks.
  • Structured Output: Generate structured outputs like JSON, XML, HTML, YAML, DSLs, etc.
  • Tool Use: Utilize external applications or APIs to enhance functionality.
  • Code Generation: Write and debug code snippets in various programming languages.
• Cross-domain use cases:
  • Summarization: Understand and condense lengthy documents into concise summaries.
  • Translation: Convert text between multiple languages with high accuracy.
  • Question Answering: Provide precise answers to user queries based on provided information.
  • Reasoning and Planning: Solve complex problems and develop step-by-step plans.
  • Personalization: Tailor responses based on user preferences and historical interactions.
  • Dialogue Management: Engage in context-aware, multi-turn conversations.
  • Data Analysis: Interpret and generate insights from structured data.
  • Content Moderation: Identify and filter inappropriate or harmful content.
  • Sentiment Analysis: Detect and interpret emotions and opinions in text.
  • Robotics Integration: Interface with robotic systems for control and decision-making.
  • Knowledge Retrieval: Access and present information from vast datasets or knowledge bases.
  • Creative Writing: Generate poetry, stories, and other creative content.
  • Educational Assistance: Provide explanations and tutoring across various subjects.
  • Ethical Reasoning: Assess scenarios for ethical considerations and implications.
  • Accessibility Support: Assist users with disabilities through tailored interactions.
  • Simulation and Modeling: Create predictive models and simulate scenarios.
• Domain-specific use cases:
  • Legal and Medical Assistance: Offer information and guidance within legal and medical domains.
  • Gaming: Generate narratives, dialogues, and scenarios for interactive entertainment.
  • Scientific Research: Aid in literature reviews, hypothesis generation, and data interpretation.
  • Financial Analysis: Analyze market trends and provide investment insights.
  • Cultural Competence: Understand and respect diverse cultural contexts in interactions.
  • Security Applications: Detect and respond to potential cybersecurity threats.
  • Environmental Monitoring: Analyze data related to environmental changes and sustainability.
  • Healthcare Support: Assist in patient monitoring, diagnostics, and personalized treatment plans.
  • Supply Chain Optimization: Enhance logistics and inventory management through predictive analysis.
  • Customer Service: Provide automated support and resolve customer inquiries.
  • Market Research: Analyze consumer behavior and market trends for business insights.
  • Content Creation: Generate articles, blogs, and marketing materials.
  • Virtual Assistance: Manage schedules, reminders, and personal tasks.
  • Social Media Management: Craft posts and engage with audiences across platforms.
  • Human Resources: Assist in recruitment, training, and employee engagement strategies.
  • Event Planning: Organize and coordinate events, including logistics and communication.
  • Travel Planning: Provide itineraries, booking assistance, and destination information.
  • Real Estate: Analyze property markets and assist in buying or selling decisions.
  • Agriculture: Monitor crop health and optimize farming practices through data analysis.
  • Energy Management: Optimize energy consumption and monitor renewable energy sources.
  • Transportation: Enhance route planning and traffic management systems.
  • Urban Planning: Assist in designing sustainable and efficient urban infrastructures.
  • Disaster Response: Provide real-time information and coordination during emergencies.
  • Public Policy: Analyze data to inform policy decisions and predict societal impacts.
  • Art and Design: Generate visual art concepts and assist in creative design processes.
  • Music Composition: Create original music pieces and assist in songwriting.
  • Language Learning: Facilitate language acquisition through interactive exercises and feedback.
  • Historical Analysis: Interpret historical data and provide insights into past events.
  • Philanthropy: Identify charitable opportunities and assess the impact of donations.
  • Sports Analytics: Analyze player performance and game strategies.
  • Fashion: Predict trends and assist in clothing design and merchandising.
  • Culinary Arts: Generate recipes and provide cooking guidance.
  • Astronomy: Analyze celestial data and assist in space exploration research.
  • Psychology: Offer insights into human behavior and mental health support.
  • Linguistics: Analyze language patterns and assist in translation studies.
  • Archaeology: Assist in artifact analysis and historical site interpretations.
  • Literature Analysis: Interpret literary works and provide critical analyses.
  • Philosophy: Engage in discussions on ethical dilemmas and existential questions.
  • Mathematics: Solve complex equations and assist in theoretical research.
  • Physics: Model physical phenomena and assist in experimental design.
  • Chemistry: Analyze chemical compounds and predict reactions.
  • Biology: Assist in genetic research and ecological studies.
  • Geology: Analyze geological data and assist in natural resource exploration.
  • Meteorology: Predict weather patterns and analyze climate data.
  • Oceanography: Study marine ecosystems and assist in ocean exploration.
  • Anthropology: Analyze cultural data and assist in ethnographic research.

• Automated geoscience image descriptions Ref
• Interpret Wind Turbine photos and charts, construction monitoring, equipment maintenance & charts Ref
• Forecast weather based on cloud photos! Ref
• Analyze thermal image of solar panels, electroluminescence images for warranty claims, ROI estimates from Google Sunroof rooftop images Ref
• Corrosion detection in electricity towers, turbines, storage tanks, penstock. Interpret non-destructive test images Ref

• Get THE domain expert (or approver) as the tester.
• Create a dataset that is DIVERSE.
• Covers EACH combination of:
  • Features
  • Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error
  • Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user
• Generate data using existing data + synthetic data for each SPECIFIC combination of the above
• Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include:
  • Nuances: Something a failed response did well or a passed response didn't quite do well
  • Improvements: Suggest how model can improve
• Build an SPA to make it easy for the domain expert to review

• Jina AI Meta Prompt is an example
• Remotion system prompt is an example
• https://docs.fastht.ml/llms-ctx.txt
• https://docs.fastht.ml/llms-ctx-full.txt

• "One of the things we learned is (an LLM app) after it passes passes frankly even 100 tests, the odds that it will do, on any random distribution of user inputs, the next 100,000 100% accurately is very high."

• o1-preview > claude-3.5 sonnet on code editing
• claude-3-opus > claude-3.5-sonnet on code refactoring
• deepseek-coder-v
• gpt-4o-mini sucks.

• Vaishnavi and Ranjeet edited code
• Indal and Koustav didn't

• E.g. Indal trying for a confetti animation, which is hard but do-able

• 04 Oct 2024. PDF-Extract-Kit does PDF layout, formula, table, and OCR extraction using various models.
• 04 Oct 2024. llmsherpa extracts PDF layout, tables, not OCR

• Claude
• OpenAI
• Gemini

• Makes you more ambitious
• Lets you code without stress. (Just pass it the error and have it fix it. Or find another approach)
• Is unlimited. You can run dozens of agents in parallel

• Suno is fast, has a better UI, lots of examples
• Udio is slow, poor UI, creates richer music

• Rouge 2 and Bleu and such metrics are NOT good. Create you own benchmarks
• Non-PEFT fine tuning needs 6X GPU RAM. Optimizer states, Gradient, Activations are the overhead. PEFT is about tuning a subset of parameters.
• LORA adds additional weights without updating the model. It's a low rank matrix multiplication. You can change these adapters in runtime. Saves space. Fast to train
• Quantization: Stick to bitsandbytes or AWQ (may be a bit better)
• QLORA = Quantization + LORA
• Predibase has open-sourced Lora Adapters in "Lora Land". Existing adapters are pretty good.
  • ghcr.io/predibase/lorax:main Docker image works on Docker compose to run locally. devices: on Docker Compose lets you specify NVIDIA GPU devices
  • Locust is a HTTP load testing lib in Python
• Techniques for inference optimization
  • Dynamic adapters: Loads right LORAX adapters WHEN a request comes in
  • Multi-adapter batching: Process all inputs in parallel on the same GPU, but different users are post-processed using different adapters

• What We’ve Learned From A Year of Building with LLMs
  • Strategy
    • IS IT TOO HARD/EXPENSIVE? Log it. LLMs are getting cheaper and better.
    • WILL OPENAI BUILD IT? If so, wait for it instead of building.
    • HAS A STARTUP BUILT IT? If so, use it instead. It's a generic use case there's no point re-inventing.
    • FOCUSED USE CASES over generic. Build trust by starting small.
    • Tools for LLM Ops (feedback): LangSmith, Log10, LangFuse, W&B Weave, HoneyHive
    • Human in the Loop is about humans evaluating model outputs. That's different from AI in the loop, human in the center, where AI accelerates human output (like Github Copilot)
  • Operations
    • CHECK EMBEDDINGS DRIFT over time. Users might be input-ing different things than before.
    • LOG AND REVIEW everything.
    • Instructor coaxes structured output from LLM APIs.
    • IMPLICIT FEEDBACK collection is easy. Just let users edit stuff.
  • Tactical
    • Try n-shot prompting (n=5-12) before bigger models.
    • Always structure for output: Markdown, XML/HTML tags.
    • Combine RAG with Keyword search. It reduces user frustration in edge cases.
    • Prefer multiple small prompts to one big prompt. Do X. Then Y. Then Z.
    • Jitter prompts for diversity beyond temperature.
    • LLM-as-judge works better when comparing outputs (not rating 1 output). Keep length similar (LLMs prefer wordiness). Swap order and compare. Allow for ties. Ask for reason FIRST.
• Hermes: A Text-to-SQL solution at Swiggy
  • "Hermes performed significantly better for charters with well-defined metadata and a relatively smaller number of tables."
  • "We collect feedback on the accuracy of the returned query from stakeholders directly within the Slack bot."
• How I use AI and "Replacing my right hand with AI"
  • EMBED in every app/workflow. E.g. Auto-fix spellings. Auto-review code. Auto-ask LLM on errors and apply patch! Auto-search for answer, assess, continue.
  • PERSIST. Stick with the LLM to the end. Don't fix it yourself. It's faster.
    • INTERVENE FAST. If an LLM can't solve it by itself in 2 tries, it needs in-depth help.
  • APP-IFY one-off tasks. Disposable tools. "Write web-app to convert JSON to tab-delimited." "Extract fields as a table." "Diff JSON."
  • BEST language/frameworks preferred. CUDA in Python. Rust. C. Raspberry Pi. Arduino. Bluetooth. Modern ESM/JS.
  • TEACH examples. "Here's the LLM Foundry API." "Here's how to use gramex.data."
  • DUMP entire code. Models can handle it. Refactoring to SQLAlchemy 2, Pandas 2. API Documentation. Test case generation.
  • ASK for features & packages. Docker without root access. GPU access inside docker. Windows CLI-only C++ compiler.
  • TEST CASE writing.
  • SPEC IN DETAIL. Use these libraries. Write like this: code example.
  • SPEC USAGE in detail.
    • "I will just pipe it into sqlite", or "I will just run ffmpeg -i filename [YOUR OPTIONS].
    • Describe the UI, API input/output, data structure, and internal data structure.
  • HELP on usage. "ffmpeg to get audio.mp3".
• My benchmark for large language models
  • LLM(text) is a useful function to have in JS and Python too. Useful as a simple pip install llmfoundry
  • Allow images, files in LLM()
• Current list of (or hard) things for LLMs
  • Translate technical documents to Dutch -- because they don't understand the technical terms well
  • Translate large documents (JSON to XML, English to Chinese, Python to Rust, Wrong to right spelling) -- because the output tokens are limited

• Defensive UX: Microsoft, Google, and Apple have guidelines for Human-AI interactions
• Collect feedback: Explicit and implicit

• LiteLLM: Open source, OpenAI compatible, 100+ LLMs
• RouteLLM: Open source, OpenAI compatible, automatically routes based on cost
• OpenRouter: OpenAI compatible API, several models
• Unify: Supports many models
• Portkey: Supports popular providers
• Martian: Limited set of models

• Databuck by FirstEigen. Autolysis plus monitoring
• Quality council has the data steward (maintainer of each dataset) coming together with the uses on a weekly basis to understand what quality problems to users are facing. Data owners jaundice at a lower frequency to get an understanding
• Automate rules for data quality in our projects and intranet
• Convert a config rule into business language. Explain SQL. These are good use cases for llm's
• Graph DBs are powerful for flexible data structures, but query generation needs AI or expertise. Check the Neo4J language cypher
• Explore storing SAME data in relational DBs AND in graph DBs / document DBs for different use cases
• Dallas rocketry challenge. Build a rocket that can take an egg to 800 feet exactly and land without breaking it

• Example of fine-tuning Mistral. Consumed 28 computes ($2.8)
• Axlotl is what the top fine-tuned LLMs are trained on
• Deepspeed provides distributed training
• Flash attention lets data stay on GPU
• Sample packing packs samples of different lengths into equal length tensors

• opengpts - but it doesn't support auth
• chatbot-ui - but Supabase is hard to install
• anse - but it doesn't support auth
• ChatGPT-Next-Web - but it doesn't support auth

• "Every text box on the internet will get an LLM"
• "Infinite interns"
• "Every UNIX function has become a company." "Every ChatGPT suggestion..."

• galactic can use LLMs to detect PII
• Presidio by Microsoft
• Sherlock is a generic sematic type matching DL model
• pii-extractor-llm was trained on Indian names
• GLiNER is a Lightweight Generalist model for NER