#code-agents

• Wed, Jan 1, 2025. LLM Code Execution Sandboxes that let you run code in a sandbox via an API: #cloud #code-agents #github #hosting #llm-ops
  • AgentRun: open source, via Docker
  • e2b.dev. A day costs about $1 (on demand) and you get about $100 one time credits. Self-hosting is complex. Discussion

• Thu, Dec 26, 2024. WOW. This is how a software agent will work alongside humans: Fix issue #5478: Add color to the line next to "Ran a XXX Command" based on return value - using @openhands-agent. #automation #code-agents #future

• Thu, Dec 26, 2024. Learnings from Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) #ai-coding-tools #code-agents #dev #github #prompt-engineering
  • Passing code execution as a tool is more powerful than granular tools. You combine multiple tools and tool calls into one. You move code to the data rather than the other way around. Mostly, you need bash, Python (or Jupyter), file manager, web browser.
  • UI: Go where the user is, instead of bringing them to you.
  • A remote runtime is a critical component.
  • Claude 3.5 Sonnet (20241022) and Claude 3.5 Haiku (20241022) perform best on SWE Bench, followed by Deepseek V3, then O1 2024-12-17. X

• Mon, Dec 23, 2024. Medical coding is an area ripe for LLMs. #ai-coding-tools #code-agents #llm-ops
  • Ojasvi Yadav created a repo that uses hierarchical classification (rather than embeddings) to find the right coding.
  • Gemini models seem to understand medical terms better than others.
  • RapidClaims, funded by TogetherAI, is apparently working on this problem.

• Sun, Dec 22, 2024. Examples of agentic AI #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
  • Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
  • Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
  • Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
  • Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
  • Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
  • Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
  • Meeting notes and action items generator: A system that generates meeting notes and action items.
  • O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
  • UI enhancement agent: An agent that added features like language locales and dark mode to a UI.

• Sat, Dec 21, 2024. Anthropic defines agents. Building effective agents + Cookbook #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
  • Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
    • Prompt chaining: Pipe each LLM output to the next LLM. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
    • Routing: One LLMs decides which other LLM to call next. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
    • Parallelize: Sectioning: Break tasks into independent subtasks, then aggregate. E.g. Evaluate contracts against different clauses in parallel.
    • Parallelize: Voting: Run same task multiple times, then vote. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
    • Orchestrator-workers: One LLM delegates to others, then syntheisizes. E.g. Search from multiple sources, then synthesize. Change multiple code files.
    • Evaluator-optimizer: One model checks another in a loop. E.g. Literary translation. Self-healing code. Policy violation checks.
    • Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages.
  • Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.

• Wed, Nov 27, 2024. Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote #automation #code-agents #prompt-engineering
  • Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks
  • Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck
  • A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once
  • Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows
  • People find it very easy to understand agentic workflows once they read through the code
  • Reflection or feedback is a useful agentic pattern
  • In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction

• Mon, Nov 25, 2024. The OpenAI Desktop App can interact with native applications, e.g. read from Terminal, VS Code, etc. This takes it on a path to becoming a copilot for ANY apps. Putting every copilot app and every LLM integration under threat. #automation #code-agents

• Mon, Nov 25, 2024. Crawl4AI and Firecrawl are tools / libraries to convert websites into LLM Friendly Markdown and extract structured data using LLMs. #ai-coding-tools #code-agents #document-conversion #html #llm-ops

• Sat, Nov 23, 2024. OpenAI lets you download GPT instructions and execute arbitrary code in their containerized environment. This is not a bug. Ref #ai-coding-tools #chatgpt #code-agents #github #prompt-engineering

• Fri, Nov 22, 2024. For good structured mode output, we need good prompting. #ai-coding-tools #code-agents #markdown #prompt-engineering
  • Mentioning examples and schema and "JSON" helps. When providing examples, using [user, assistant] message pairs helps (I think it's because it's easier for the LLM to parse).
  • Using a {reasoning, answer} schema (with reasoning first) helps. Make reasoning concise and relevant Ref Arxiv
  • We already know code in JSON is not a great idea. Ref

• Tue, Nov 19, 2024. Langchain suggests multiple levels of agentic behaviour. LLM Call < LLM Chain < LLM Rounter < State Machine < Autonomous Langchain #code-agents #llm-ops

• Sat, Nov 9, 2024. Google counts auto-completion when saying "25% of all the code is written by AI at Google". "It's a helpful productivity tool but it's not doing any engineering at all. It's probably about as good, maybe slightly worse, than Copilot." YCombinator #ai-coding-tools #code-agents #prompt-engineering

• Fri, Nov 8, 2024. Here are some approaches to large-scale classification of medical codes. ChatGPT #ai-coding-tools #code-agents
  • Fine-Tuning LLMs on Medical Data: Enhance LLMs by training them on medical datasets, such as clinical notes and discharge summaries, to improve their understanding of medical terminology and context.
  • Multi-Agent Frameworks: Implement a multi-agent system that simulates real-world coding processes with distinct roles (e.g., patient, physician, coder, reviewer, adjuster). Each agent utilizes an LLM to perform specific functions, enhancing interpretability and reliability. ArXiv
  • Retrieve-Rank Systems: Develop a two-stage system where the LLM first retrieves potential ICD-10 codes and then ranks them based on relevance, improving precision in code assignment. ArXiv
  • Embedding-Based Approaches: Use LLMs to generate embeddings for ICD-10 codes and medical texts, facilitating the matching of texts to appropriate codes through similarity measures. GitHub
  • Hierarchical Classification: Leverage the hierarchical structure of ICD-10 codes by first classifying texts into broader categories before assigning specific codes, reducing complexity and improving accuracy. ArXiv
  • Two-Stage Verification Models: Combine LLMs with verification models, such as Long Short-Term Memory (LSTM) networks, to validate and refine the codes suggested by the LLM, balancing recall and precision. ArXiv
  • Also, a mixture of models approach might work. Feed any existing NLP model / rules as a second opinion.

• Mon, Nov 4, 2024. Zed.dev is an AI code editor by the creators of Atom. It's written in Rust and is blazing fast. It has native AI integration. #ai-coding-tools #code-agents #dev #github #markdown

• Mon, Nov 4, 2024. Chain of Thought reduces performance where thinking makes humans worse. Ref. Specifically: #code-agents
  • Artificial grammar learning
  • Facial recognition
  • Classifying data that has exceptions

• Mon, Oct 28, 2024. Pragmatic Podcast (I forgot which one) #ai-coding-tools #automation #code-agents #future #github #prompt-engineering
  • Automate changelogs for your codebases. Convert past commits into attractive release notes automatically
  • AI is going to be the consumer of many tools and logs. Build converters for these
  • Speed of validation such as linting, testing, etc. will allow LLMs to iterate faster and WILL become more important

• Sun, Oct 27, 2024. AI code editors are not good with large code bases today. Keep the refactoring exercises to below 1000 lines. Also evaluate the ease of setting it up locally #ai-coding-tools #code-agents #github #markdown

• Thu, Oct 24, 2024. Gemini has a code execution API! #ai-coding-tools #code-agents

• Sun, Oct 6, 2024. Continue.dev is another AI coding editor. It supports OpenRouter. So now I have heard good things about: #ai-coding-tools #code-agents #dev #github #prompt-engineering
  • Github Copilot
  • Cursor
  • Cody
  • Continue.dev (supports OpenRouter)
  • Aider (supports OpenRouter)
  • Maybe:
    • Codeium
  • Not:
    • Amazon Q Developer

• Thu, Oct 3, 2024. Quotes on Graph RAG from a Generative AI WhatsApp Group. #code-agents
  • "Damn so literally nobody uses Graph RAG yet. Good to know." ~Sumba
  • "A big four consulting firm uses GraphRAG to retrieve related documents and excerpts from governance and compliance docs." ~Vinayak Hegde (Microsoft)
  • "Graph RAG is expensive and unnecessary in most of the cases." ~Utkarsh Saxena

• Mon, Sep 30, 2024. Coders are more likely to edit LLM code. Non-coders don't have that bad habit. #ai-coding-tools #code-agents #dev #github #llm-ops #markdown
  • Vaishnavi and Ranjeet edited code
  • Indal and Koustav didn't

• Fri, Sep 20, 2024. Prompt engineering for code generators: #ai-coding-tools #code-agents #dev #future #markdown #prompt-engineering
  • Claude Artifacts Prompt
  • Val.Townie system prompt. Good example of how to create
  • Cursor editing
  • Cursor debugging
  • Cursor conversation

• Fri, Sep 20, 2024. XML tags seem best to structure prompts across LLMs. #ai-coding-tools #code-agents #llm-ops #markdown #prompt-engineering
  • Claude
  • OpenAI
  • Gemini

• Sun, Sep 15, 2024. Prompt engineering tip: Tell LLMs another AI wrote code. Else they will agree with you! #ai-coding-tools #automation #code-agents #future #llm-ops #prompt-engineering

• Wed, Sep 11, 2024. LLM coding #ai-coding-tools #code-agents #future #llm-ops
  • Makes you more ambitious
  • Lets you code without stress. (Just pass it the error and have it fix it. Or find another approach)
  • Is unlimited. You can run dozens of agents in parallel

• Sun, Sep 8, 2024. The Impact of AI on Computer Science Education: #ai-coding-tools #code-agents
  • Eric Klopfer divided his undergrad CS class into three groups and gave them a Fortran task.
  • One used ChatGPT. Another, Meta's Code Llama LLM. Third, only use Google.
  • ChatGPT group was faster than Code Llama was faster than Google
  • When tested on the approach, the ChatGPT remembered nothing. Half the Code Llama group passed. The Google group passed fully

• Sun, Sep 8, 2024. Via Pratap Vardhan: #ai-coding-tools #code-agents #github
  • At Khan Academy, every developer working on Khanmigo has cursor. Everyone who's contributed to a Khan Academy GitHub repo has GitHub Copilot.
  • I stopped using Google + StackOverflow 2 years ago. I use ChatGPT, Copilot, etc. For humans, I ask Reddit.
  • Excited by async agents. Things that do my job while I sleep.
    • Zapier notifications.
    • Monitor what happens. Put it into a flow diagram and alert me.
    • Every month, did my broker trade? Did my bank transaction fail? Did I pay my electricity bill?
    • Every time you delegate, use an agent instead.
    • Read my RSS feeds.
    • Read my browser history and suggest interests.
  • Plan a session in Bain, BCG, etc. on Artifacts.
  • Explore sparse embeddings. More effective. ColiPali, ColBERT

• Fri, Aug 23, 2024. An 8-year old uses Cursor.ai to code #ai-coding-tools #code-agents #future

• Mon, Aug 19, 2024. Copilot's system prompt calls a search_enterprise(query: str) tool and a hint(M365Copilot_language: str) tool as assistants. #code-agents #prompt-engineering

• Mon, Aug 12, 2024. Code agent frameworks to explore: #ai-coding-tools #automation #code-agents #dev #future #github #markdown #prompt-engineering
  • Cognition
  • Factory
  • Codegen

• Thu, Aug 8, 2024. Agentic RAG (Ravi Theja, LlamaIndex) #ai-coding-tools #code-agents
  • RAG via top-k retrieval fails with
    • summarization => need to read all chunks
    • comparison: compare product X vs Y => need to split and re-combine
    • structured analytics. e.g. most expensive employees => Text2SQL first
    • multi-part questions. e.g. Tell me about speed of model X AND cost of model Y and recommend => need to split and re-combine
  • RAG failures: It's single shot. No query planning. No tools. No correction. No memory.
  • Agents that help in RAG
    • Route to the right tool
      • E.g. retrieve via vector top-k search or vector summary search or keyword search or combination?
    • One-shot query planning
      • E.g. Break query into multiple specific queries. RAG those. Then combine. #TRY - maybe in DocSearch
    • Tool use
      • E.g. Schema retrieval, Text2SQL, Calendar, Chat, APIs, Search, etc.
  • Agent orchestration
    • ReAct: An agent reasoning loop. Reason + Act. {Thought, Action, Action Input, Observation}*.
      • Orchestrate tools with a prompt
    • Multi-agent task solver: Llama agents
      • Instead of a single agent loop, use different agents. Also allows parallelization
      • Allow services to register. (MS TaskWeaver stores tool descriptions in YAML)
  • LlamaHub Tools has ideas for agents

• Wed, Aug 7, 2024. micro-agent generates test cases first when asked to build an app. Then it iterates until the test cases pass. #code-agents #future #github

• Sun, Jul 14, 2024. Github Copilot internals walks through how Copilot constructs its prompts #code-agents #future #prompt-engineering

• Wed, May 29, 2024. AI reporting templates are a trend. Domain expertise comes in via structuring the report template and associated prompts. #ai-coding-tools #automation #code-agents

• Tue, May 28, 2024. #POST I tried Undermind.ai - an agent that researches for you. It guides you to ask a detailed question, spends 2-3 minutes finding the answer, and provides detailed results. But it's worth the wait. It's a good alternative to quick validations on SciSpace. #ai-coding-tools #code-agents #future

• Sun, Apr 14, 2024. With RAG on search becoming big, search APIs are growing. serper.dev, you.com, searxng being examples. #ai-coding-tools #code-agents #search

• Thu, Apr 11, 2024. VS Code #ai-coding-tools #code-agents #github #markdown
  • Select any code and command Explain this to understand the code
  • %something in command bar searches ACROSS files for a term. Exactly like Ctrl+Shift+F
  • Copilot has an Inline Chat: Start in Terminal (that needed me to unbind Ctrl+I in bash to work)
  • Ctrl+2 opens a second window on the side. Ctrl+1 goes back to the first window
  • Terminal: Open Detected Link lets you scroll through detected (file) links in terminal
  • Terminal sticky scroll is transparent. (But Terminal stick scroll isn't working for me.)
  • Copilot uses last 10 commit messages, Jupyter notebook kernel state (variables) as additional context
  • 1.88: supports locked scrolling to sync scrolling of side-by-side windows

• Sat, Mar 30, 2024. browser.new is a good example of a browser agent. It slowly but independently does a good job of achieving the result. Example: What crew is common in Ingrid Bergman - Cary Grant films? #code-agents #prompt-engineering

• Wed, Feb 28, 2024. SourceGraph is the default platform for private code completion & search #ai-coding-tools #code-agents #future #github #markdown

• Sun, Feb 11, 2024. Two pass generation is a clever technique to get multiple SEQUENTIAL answers in a single API request. For example the schema {'code', 'optimized_code'} will generate code and then optimize it. #ai-coding-tools #code-agents #future #optimization #prompt-engineering

• Sun, Jan 28, 2024. AppAgent is an LLM that can navigate mobile / web apps #code-agents #future #llm-ops

• Sun, Jan 28, 2024. Retrieval Centric Generation is an emerging alternative to RAG, where the LLM is explicitly built to leverage external knowledge. SimplyRetrieve is an early implementation. #code-agents #future

• Sun, Jan 7, 2024. CodeGPT is an alternative to Github Copilot that can use any LLM. #ai-coding-tools #chatgpt #code-agents #future #github #llm-ops

• Mon, Dec 11, 2023. ast-grep is a semgrep alternative that focuses on code refactoring rather than security. Comby is another such tool #ai-coding-tools #code-agents #github

• Thu, Dec 7, 2023. CodeAnt.ai is a VS Code plugin to detect code smells, refactor for modularity, to write docstrings and unit tests #ai-coding-tools #code-agents #github #markdown

• Thu, Nov 30, 2023. CoVA scrapes web pages via OCR #ai-coding-tools #code-agents #document-conversion

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding-tools #automation #code-agents #future #llm-ops #optimization #prompt-engineering

• Sat, Nov 25, 2023. Quote from Jerry Liu: "RAG is a hack". It's engineered, not machine learnt, so it's suboptimal. We need an ML way of creating the context. Maybe fine tuning can be a way of CREATING the right context. But RAG can handle deterministic stuff like access control. #ai-coding-tools #code-agents #gpu

• Sat, Nov 25, 2023. Agents mimic programming primitives. Switch. While. Call a function. Print. #ai-coding-tools #automation #code-agents #prompt-engineering

• Tue, Nov 21, 2023. CoVA scrapes web pages using LLMs and visual information. #ai-coding-tools #code-agents

• Mon, Nov 6, 2023. microsoft/autogen enables multi-agent conversations. #automation #code-agents #future

• Mon, Nov 6, 2023. 06 Nov 2023: ChatGPT is slightly better that Github Copilot or CodeWhisperer #ai-coding-tools #chatgpt #code-agents #github

• Sat, Nov 4, 2023. 04 Nov 2023: Running LLMs in production lists experiments to run. Validation CAN be automated! #code-agents #llm-ops

• Thu, Oct 26, 2023. 26 Oct 2023: ChatGPT AutoExpert is a great prompt mechanism for higher accuracy, context and control. Example #ai-coding-tools #automation #chatgpt #code-agents #prompt-engineering #tts