#automation

• Fri, May 23, 2025. "Agents are models using tools in a loop." -- Hannah Moran Simon Willison #ai-coding-tools #automation #code-agents #future #models

• Mon, May 12, 2025. We could "fine-tune" system prompts automatically with evals, creating a "system prompt learning" paradim -- like my promptevals. Andrej Karpathy #ai-coding-tools #automation #future #markdown #optimization #prompt-engineering

• Mon, May 12, 2025. Model reliability is a huge enabler for performance. As models become more reliable, they can work autonomously for longer and that is another kind of scaling. Vending Bench #automation #future #gpu #models

• Thu, May 8, 2025. Double-checking is surprisingly efficient since LLM hallucinations are mostly uncorrelated. LLMs perform human tasks (e.g. classifying customer support messages) at ~85% accuracy. This might be unacceptable. But by asking 2 moderately correlated LLMs and double-checking discrepancies, we reduce automation by ~20% but reduce errors to 0.25%. Triple-checking reduces automation by ~25% but errors to under ~0.01%! Ref #automation #code-agents #llm-ops

• Tue, Apr 29, 2025. Notes from AI Snake Oil #automation
  • PCs took 3 years to hit 20% of US population. ChatGPT took 2 years for 40%. But it's a lot cheaper, and a lot less used (0.5-3.5% of work hours). Maybe Gen AI adoption is slower than PCs.
  • The jagged edge of capability: some things will become MUCH easier while others don't. The relative mix determines who goes out of a job and which tasks get fully automated. Benchmarks are rare in areas where AI is weak.
  • Factory electrification took 40 years - to redesign the layout & process; change the org structure & policies; hiring & training practices. AI diffusion could take as long.
    • Therefore, the ability to re-structure a workflow end-to-end will be an advantage.
  • Several areas of low AI capability will improve slowly because the feedback is slow due to safety regulations, human adoption speed, lack of clarity on what is better, slow physical feedback (e.g. growing trees), etc.
  • Human intelligence is in the use of technology. AI is one more such technology.
  • We know of good system safety controls in complex systems like aircrafts, power grids, engineering, chip design, healthcare, cyber-security, etc. Circuit-breakers, predefined rules, audits & monitors, access control, formal verification, etc.
  • Even if everything humans do TODAY is automated, it doesn't mean we won't have work. It just shifts to what we're not doing today.
    • We stopped work 4,000 years ago, with the agricultural revolution. The plant/livestock does all the growing. We just manage them, moving stuff around.
    • We stopped work 400 years ago, with the industrial revolution. Machines do the moving. We just manage them, computing the moves.
    • We stopped work 40 years ago, with the information revolution. Computers do the computation. We just manage them, thinking how.
    • Most future tasks will be managing AI that do the thinking.

• Sun, Apr 27, 2025. With LLMs writing code, it's becoming practical to run so many more things in SQL -- such as parsing HTML. Simon Willison #ai-coding-tools #automation #code-agents #dev #future #html #llm-ops #markdown #prompt-engineering

• Tue, Apr 22, 2025. If AI interacts with the world and generates data from its own experience and learns from that, we have a new scaling mechanism. DeepMind podcast #automation #future

• Mon, Apr 21, 2025. Good managers will probably make good AI managers. AI agents can probably substitute humans in business experiments. Ethan Mollick #automation #future

• Sat, Apr 19, 2025. AI coding can teach us new tech. For example I learned that tqdm.pbar can print logs while showing progress. It's worth noting such learnings until it becomes a habit. #ai-coding-tools #automation #code-agents #future #github #markdown #prompt-engineering

• Fri, Apr 18, 2025. O3 and O4 have built-in tool use covering all of OpenAI's tools, including containers. This allows them to manipulate images and natively understand them improving vision capabilities dramatically. #ai-coding-tools #automation #gpu

• Tue, Apr 1, 2025. Zapier has an MCP server. That should make most integrations easier. #automation #future

• Tue, Apr 1, 2025. Airflow AI SDK is a clever idea. Airflow is a workflow system. Agents are a workflow system (sort of). This SDK exposes LLMs as Airflow tasks. #automation #code-agents #future #llm-ops

• Sat, Mar 29, 2025. Reflecting on Satya Nadella's "SaaS is dead", building or porting apps' functionality into classic chatbots (e.g. via MCPs) would be an emerging market. E.g. "Create a HubSpot MCP. Do whatever you want on HubSpot, except via ChatGPT or your favorite LLM chatbot." To be fair, such interfaces exist. #automation #prompt-engineering
  • HubSpot MCP with a vega-lite MCP and a few others could solve many common HubSpot UI tasks.
  • DarwinBox MCP, ZenDesk MCP, etc. are emerging.

• Sat, Mar 15, 2025. Playbook driven agents are another approach to agentic workflows. Simon Willison #agents #automation #code-agents #future

• Fri, Mar 14, 2025. Unsloth might be one of the simplest ways of fine-tuning. #automation #markdown #optimization #prompt-engineering

• Fri, Mar 14, 2025. Apify is a cloud scraper platform. Here's how they optimize their AI Web agent - Source: #ai-coding-tools #automation #code-agents #html #optimization #prompt-engineering #web-dev
  • Remove redundant tags and attributes (e.g. accessibility, etc.). Explore readability.
  • Add a unique gid to each element.
  • Add the screenshot WITH a "Set of Marks" - "SoM" (read research paper) highlighting important clickable elements.
  • Code output is brittle. Use tools / DSL - e.g. visit_url(url), click_element(text, gid, tagName), etc.

• Fri, Mar 14, 2025. GenAIScript increasingly looks like a promising way to automate LLM workflows in the browser. #ai-coding-tools #automation #code-agents #future #github #llm-ops

• Wed, Mar 5, 2025. Build apps (not just content) for agents. In the next 3 to 5 years, agents will surpass humans as the top product users. #agents #automation #code-agents #future

• Tue, Mar 4, 2025. Tools and solutions should fit within existing workflows. That means almost all capabilities need to be exposed as APIs. #automation #future #prompt-engineering

• Mon, Mar 3, 2025. Notes from AI Engineering Summit, NY, Day 1 #ai-coding-tools #automation #code-agents #prompt-engineering
  • An agent requires 3 things: a router, tools or skills, and memory.
  • Agents are often sequential, but sometimes parallel execution makes sense for independent tasks that you consolidate.
  • Always allow LLMs the option of NOT answering a question if there is no good answer.
  • Focus prompts on the happy path. Use guard rails for edge cases.
  • Here are a few "tools" an agent would need to call:
    • Clarification from user
    • Saving to memory
    • Google search
    • Edit a file introducing SPECIFIC changes
    • Search in codebase using embeddings
    • Run scripts on the shell or in a REPL (Python, Node, etc.)
    • Run code in a new container for isolation
    • Automatically discover, read an API documentation and use it
  • Modify environment to enable logging and other system changes.
  • When code is cheap, you can explore more ideas and hence design and product management need to approach things differently. We also need to reaching testing completely because it makes very different kinds of mistakes and we don't often have an intuition
  • You can have an agent explore all the issues and full request and recent comments against the repository and summarise it for the project manager

• Mon, Mar 3, 2025. Notes from AI Engineering Summit, NY. Session by Lux Capital. #automation #code-agents #llm-ops #optimization #prompt-engineering
  • Agents make multiple LLM calls. Errors accumulate. So the quality of the model is key
  • What's really critical: data + context + user preference
  • Set up evals for subjective responses by collecting signals continuously.
  • Create scaffolding for agents where errors don't accumulate. Better yet, make it FIX errors
  • UX is critical. We need lots more UX styles

• Fri, Dec 27, 2024. Satya Nadella prepares for meetings by asking Copilot to tell him everything he needs to know about the client from the CRM, emails, meeting transcripts etc. He shares that colleagues who annotate it further for him. That's using AI for reasoning and collaborating with colleagues. Satya Nadella | BG2 w/ Bill Gurley & Brad Gerstner #automation #prompt-engineering

• Thu, Dec 26, 2024. WOW. This is how a software agent will work alongside humans: Fix issue #5478: Add color to the line next to "Ran a XXX Command" based on return value - using @openhands-agent. #automation #code-agents #future

• Sun, Dec 22, 2024. Examples of agentic AI #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
  • Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
  • Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
  • Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
  • Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
  • Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
  • Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
  • Meeting notes and action items generator: A system that generates meeting notes and action items.
  • O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
  • UI enhancement agent: An agent that added features like language locales and dark mode to a UI.

• Sat, Dec 21, 2024. Anthropic defines agents. Building effective agents + Cookbook #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
  • Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
    • Prompt chaining: Pipe each LLM output to the next LLM. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
    • Routing: One LLMs decides which other LLM to call next. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
    • Parallelize: Sectioning: Break tasks into independent subtasks, then aggregate. E.g. Evaluate contracts against different clauses in parallel.
    • Parallelize: Voting: Run same task multiple times, then vote. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
    • Orchestrator-workers: One LLM delegates to others, then syntheisizes. E.g. Search from multiple sources, then synthesize. Change multiple code files.
    • Evaluator-optimizer: One model checks another in a loop. E.g. Literary translation. Self-healing code. Policy violation checks.
    • Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages.
  • Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.

• Thu, Dec 12, 2024. I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: #automation #best-practices #llm-ops #optimization #prompt-engineering
  • Ask better questions. (Prompt engineering.)
  • Create templates or step-by-step instructions. (Chain of Thought.)
  • Ask for multiple options and pick from the best options. (Agentic approach?)
  • Training. (Fine tuning.)
  • Price weaker responses lower. (Stratified model pricing?)

• Wed, Nov 27, 2024. OpenAI allows multiple system messages. I learned this browsing through the LIDA prompts. #automation #future #llm-ops

• Wed, Nov 27, 2024. Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote #automation #code-agents #prompt-engineering
  • Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks
  • Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck
  • A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once
  • Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows
  • People find it very easy to understand agentic workflows once they read through the code
  • Reflection or feedback is a useful agentic pattern
  • In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction

• Mon, Nov 25, 2024. The OpenAI Desktop App can interact with native applications, e.g. read from Terminal, VS Code, etc. This takes it on a path to becoming a copilot for ANY apps. Putting every copilot app and every LLM integration under threat. #automation #code-agents

• Mon, Nov 11, 2024. Pass a complex data structure to Claude.ai and have it create an app to visualize it. It does well. Simin Willison #automation #markdown

• Sat, Nov 9, 2024. Zapier Actions are an easy way to set up custom actions like GMail / Google Calendar APIs for GPTs, since GPTs' callback URLs keep changing. But they fail often, and don't work on mobile. At least for me. #automation #chatgpt

• Mon, Oct 28, 2024. Pragmatic Podcast (I forgot which one) #ai-coding-tools #automation #code-agents #future #github #prompt-engineering
  • Automate changelogs for your codebases. Convert past commits into attractive release notes automatically
  • AI is going to be the consumer of many tools and logs. Build converters for these
  • Speed of validation such as linting, testing, etc. will allow LLMs to iterate faster and WILL become more important

• Thu, Oct 17, 2024. AI seems to be slowing down apprenticeship since experts would rather use an AI than train an apprentice. Example: Robotic surgery. Ref #automation #future

• Fri, Oct 4, 2024. Software 2.0 by Andrej Karpathy. #ai-coding-tools #automation #dev #markdown #prompt-engineering
  • This is fundamentally altering the programming paradigm by which we iterate on our software, as the teams split in two:
    • the 2.0 programmers (data labelers) edit and grow the datasets, while
    • a few 1.0 programmers maintain and iterate on the surrounding training code infrastructure, analytics, visualizations and labeling interfaces.
  • Adaptive UI ideas:
    • Adaptive Fields: Show only required fields based on what the user field so far.
    • Smart Inputs: Dropdowns and auto-complete based on user's context.
    • Smart Themes: Change font size, contrast, theme guessing the user's age and preferences.
    • Dynamic Menus: Show what they might need to do next. Like Nokia's right button, but using LLMs.
    • Smart Tooltips: Check what the user's doing (delays, confusions, previous clicks, current actions) and show relevant tips.
    • Personalized Layout: Show only the relevant sections of the app. E.g. based on what they're doing.
    • Smart Charts: Create the right chart that solve the user's question.
  • Adaptive Back-end
    • Dynamic APIs: Create endpoints on the fly based on user needs
    • Dynamic Indexing: Create & update indices on the fly based on user needs
    • Dynamic Schema: Create & update schema on the fly based on user needs
    • Dynamic Migration: Migrate to a new database or OS or language as required
    • Dynamic Queries: Create SQL/NoSQL queries to solve the user problem
    • Dynamic RBAC: Figure out who needs permissions and why. Add OR REMOVE access as required
    • Dynamic Logging. Log what's required. Explain why it's logged and what's happening. Fix code that raised the error
    • Dynamic Caching. Cache what's likely to be required. Evict what may not be required. Figure out cache keys.

• Sun, Sep 15, 2024. Prompt engineering tip: Tell LLMs another AI wrote code. Else they will agree with you! #ai-coding-tools #automation #code-agents #future #llm-ops #prompt-engineering

• Mon, Aug 12, 2024. Code agent frameworks to explore: #ai-coding-tools #automation #code-agents #dev #future #github #markdown #prompt-engineering
  • Cognition
  • Factory
  • Codegen

• Tue, Jun 25, 2024. LLM Patterns include Evals, RAG, Fine-tuning, Caching, Guardrails, Defensive UX, Collect feedback. Notably: #ai-coding-tools #automation #best-practices #future #llm-ops #optimization #prompt-engineering
  • Defensive UX: Microsoft, Google, and Apple have guidelines for Human-AI interactions
  • Collect feedback: Explicit and implicit

• Wed, May 29, 2024. AI reporting templates are a trend. Domain expertise comes in via structuring the report template and associated prompts. #ai-coding-tools #automation #code-agents

• Tue, Mar 19, 2024. Control vectors in real-time lets you control response in real-time #automation #future #prompt-engineering

• Fri, Feb 23, 2024. Teknoturf is using Gen AI to #ai-coding-tools #automation #prompt-engineering #speech-to-text #tts
  • Improve prompts when teaching prompt engineering.
  • Pronounce languages better, identifying which words Tamilians and Malayalis will mis-pronounce.

• Sun, Feb 4, 2024. #PREDICTION As automation takes over these mainstream activities, people will take over the niches. Since expertise like knowledge is fractal, there will be many more segments of one in the future and it will be easier to automate clusters of similar abilities. Recommenders and brands will become even more important #automation #future

• Sat, Jan 27, 2024. ⭐ OpenAI's prompt engineering strategies are an excellent start for prompt engineering. A few lessons: #automation #prompt-engineering
  • Use detailed system prompts, often containing the entire instruction set, if it won't change over the course of a conversation.
  • "... summary of the prior conversation could be included as part of the system message" is an interesting history compression tactic.

• Sun, Jan 14, 2024. AI by API is the norm today and will grow #ai-coding-tools #automation #future
  • Just having AI is no longer a differentiator
  • AI is part of life, not just work
  • #TODO Explore quickdrop from Stability for Maruti
  • #TODO Explore Codium VS Code plugin and Continue.dev
  • Hybrid systems that combine stats, ML, DL and AI models will grow
  • AGI and AutoGPT resurgence
  • RAG will continue to be a focus
  • GPT4 will be beaten by open source models. Special purpose models beat it already
  • Self hosted and cloud hosted models will grow for security
  • Small language models will grow
  • Productivity will be enhanced rather than replaced
  • Multi modal models will grow
  • Cost efficiency will grow in focus

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding-tools #automation #code-agents #future #llm-ops #optimization #prompt-engineering

• Sat, Nov 25, 2023. Agents mimic programming primitives. Switch. While. Call a function. Print. #ai-coding-tools #automation #code-agents #prompt-engineering

• Mon, Nov 6, 2023. microsoft/autogen enables multi-agent conversations. #automation #code-agents #future

• Thu, Oct 26, 2023. 26 Oct 2023: ChatGPT AutoExpert is a great prompt mechanism for higher accuracy, context and control. Example #ai-coding-tools #automation #chatgpt #code-agents #prompt-engineering #speech-to-text #tts