#prompt-engineering

• Sat, Jan 4, 2025. mitmproxy is an open source tool to intercept, modify, and replay HTTP requests. An alternative to Charles, Fiddler, and partly WireShark. Guide. Like the others, it requires installing a trusted root certificate on your machine. #ai-coding-tools #prompt-engineering
  • mitmproxy2swagger digs through the mitmproxy flows and generates an OpenAPI schema. A clever idea to reverse-engineer APIs.

• Thu, Dec 26, 2024. Learnings from Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) #ai-coding-tools #code-agents #dev #github #prompt-engineering
  • Passing code execution as a tool is more powerful than granular tools. You combine multiple tools and tool calls into one. You move code to the data rather than the other way around. Mostly, you need bash, Python (or Jupyter), file manager, web browser.
  • UI: Go where the user is, instead of bringing them to you.
  • A remote runtime is a critical component.
  • Claude 3.5 Sonnet (20241022) and Claude 3.5 Haiku (20241022) perform best on SWE Bench, followed by Deepseek V3, then O1 2024-12-17. X

• Sun, Dec 22, 2024. How are software engineers are future-proofing their careers in the face of LLMs? #llm-ops #prompt-engineering
  • Leveraging LLMs as Force Multipliers
    • Use LLMs for repetitive tasks, rapid prototyping, exploring multiple approaches, data extraction and brainstorming, providing feedback.
    • Explore prompting techniques, integrate LLMs into their workflows, and develop strategies for validating and refining LLM-generated code
  • Focusing on higher-level skills that llms struggle with
    • Systems Thinking and Architecture: code readability, extensibility, testability, and maintainability
    • Problem Solving and Critical Thinking: define problems clearly, break them down into manageable parts, and reason through complex scenarios. LLMs produce plausibly incorrect code.
    • Communication and Collaboration
    • Domain Expertise
  • Exploring Adjacent Roles: product management, technical leadership, or consulting. Involve more interaction with clients and stakeholders.
  • Developing "Evergreen" Skills: debugging, system administration, and security. Or outside of software engineering, such as trades or other hands-on vocations.
  • Scepticism: LLMs may not reach a level of sophistication that would render their expertise obsolete. Complex problems, understanding context, and producing high-quality, maintainable code.

• Sun, Dec 22, 2024. Examples of agentic AI #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
  • Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
  • Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
  • Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
  • Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
  • Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
  • Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
  • Meeting notes and action items generator: A system that generates meeting notes and action items.
  • O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
  • UI enhancement agent: An agent that added features like language locales and dark mode to a UI.

• Sat, Dec 21, 2024. Anthropic defines agents. Building effective agents + Cookbook #ai-coding-tools #automation #code-agents #llm-ops #prompt-engineering
  • Augmented LLMs are LLMs enhanced with augmentations such as retrieval, tools, and memory.
  • Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
    • Prompt chaining: Pipe each LLM output to the next LLM. E.g. Write report, then translate. Extract results, then verify them. Successively ask follow-up questions.
    • Routing: One LLMs decides which other LLM to call next. E.g. Evaluate complexity, then pick the right model. Classify request time, then pick the right prompt.
    • Parallelize: Sectioning: Break tasks into independent subtasks, then aggregate. E.g. Evaluate contracts against different clauses in parallel.
    • Parallelize: Voting: Run same task multiple times, then vote. E.g. Review code for prompt injection using different prompts. Evaluate content safety with different thresholds.
    • Orchestrator-workers: One LLM delegates to others, then syntheisizes. E.g. Search from multiple sources, then synthesize. Change multiple code files.
    • Evaluator-optimizer: One model checks another in a loop. E.g. Literary translation. Self-healing code. Policy violation checks.
    • Human-in-the-loop Checkpoints: The workflow explicitly requests human review at certain stages.
  • Agents are LLMs that dynamically direct their own processes and tool usage, consulting tools or the user as needed.

• Thu, Dec 12, 2024. I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses: #automation #best-practices #llm-ops #optimization #prompt-engineering
  • Ask better questions. (Prompt engineering.)
  • Create templates or step-by-step instructions. (Chain of Thought.)
  • Ask for multiple options and pick from the best options. (Agentic approach?)
  • Training. (Fine tuning.)
  • Price weaker responses lower. (Stratified model pricing?)

• Thu, Nov 28, 2024. HTMX focuses on HTML over JS. Like server responses being HTML snippets not JSON. But I need front-end over back-end. Client side apps. HTMX doesn't help much there, e.g. templating, or just plain JS code. #html #prompt-engineering #web-dev

• Wed, Nov 27, 2024. Andrew Ng Explores The Rise Of Al Agents And Agentic Reasoning | BUILD 2024 Keynote #automation #code-agents #prompt-engineering
  • Innovation is now a path to production. People are able to build 20 prototypes at the cost of one and see which sticks
  • Machine learning is much faster. Things that took months now date days. But engineering and evaluations are only slightly faster and have become a bottleneck
  • A good analogy to zero shot prompting is to ask a person to write an entire essay without pressing backspace even once
  • Andrew scenes to align with the line chain definition of agentic workflow, which is about agents being able to craft their own control flows
  • People find it very easy to understand agentic workflows once they read through the code
  • Reflection or feedback is a useful agentic pattern
  • In multi-agent collaboration, it may be the same underlying model that is acting as different agents. But just like we find it useful for the same CPU to run multiple processes and each application is its own abstraction, agents of useful abstraction

• Sat, Nov 23, 2024. OpenAI lets you download GPT instructions and execute arbitrary code in their containerized environment. This is not a bug. Ref #ai-coding-tools #chatgpt #code-agents #github #prompt-engineering

• Fri, Nov 22, 2024. For good structured mode output, we need good prompting. #ai-coding-tools #code-agents #markdown #prompt-engineering
  • Mentioning examples and schema and "JSON" helps. When providing examples, using [user, assistant] message pairs helps (I think it's because it's easier for the LLM to parse).
  • Using a {reasoning, answer} schema (with reasoning first) helps. Make reasoning concise and relevant Ref Arxiv
  • We already know code in JSON is not a great idea. Ref

• Fri, Nov 22, 2024. Just adding 3 real examples and regurgitation helped GPT 4o play chess much better. Both techniques may have more general use in prompting. Simon Willison #prompt-engineering

• Sat, Nov 9, 2024. Google counts auto-completion when saying "25% of all the code is written by AI at Google". "It's a helpful productivity tool but it's not doing any engineering at all. It's probably about as good, maybe slightly worse, than Copilot." YCombinator #ai-coding-tools #code-agents #prompt-engineering

• Mon, Nov 4, 2024. Creating a LLM-as-a-Judge That Drives Business Results by Hamel Husain. #best-practices #dev #llm-ops #prompt-engineering
  • Get THE domain expert (or approver) as the tester.
  • Create a dataset that is DIVERSE.
  • Covers EACH combination of:
    • Features
    • Scenarios: e.g. multiple matches, no match, ambiguous request, invalid/incomplete input, unsupported feature, system error
    • Persona: e.g. new user, expert user, non-native speaker, busy professional, technophobe, elderly user
  • Generate data using existing data + synthetic data for each SPECIFIC combination of the above
  • Evaluate based only on PASS/FAIL with a CRITIQUE detailed enough for a new employee. Include:
    • Nuances: Something a failed response did well or a passed response didn't quite do well
    • Improvements: Suggest how model can improve
  • Build an SPA to make it easy for the domain expert to review

• Mon, Oct 28, 2024. Pragmatic Podcast (I forgot which one) #ai-coding-tools #automation #code-agents #future #github #prompt-engineering
  • Automate changelogs for your codebases. Convert past commits into attractive release notes automatically
  • AI is going to be the consumer of many tools and logs. Build converters for these
  • Speed of validation such as linting, testing, etc. will allow LLMs to iterate faster and WILL become more important

• Mon, Oct 28, 2024. GH CLI is quite good for deployment too, like Wrangler CLI. Enabling pages, setting secrets, etc. #dev #github #prompt-engineering

• Mon, Oct 14, 2024. How AI can improve education performance and engagement. Ref #ai-coding-tools #future #prompt-engineering
  • Student: Create study plan based on course and schedule
  • Student: Focus on what you need to learn more
  • Student: Align with your study style and pace
  • Teacher: Grading MCQs
  • Teacher: Writing conceptual guides
  • "New collar workers" was coined by Ginny Rometty
  • Embed tutor or document in video and ask for clarification! This is a new embedded interface. #TODO

• Sun, Oct 13, 2024. OpenAI has a prompt generator. Currently it uses a meta-prompt but may later move to DSPy or Gradient Descent. Ref #prompt-engineering

• Wed, Oct 9, 2024. O1 is good at solving problems where the solution is easy to verify and generating options helps get closer to the solution #future #llm-ops #optimization #prompt-engineering

• Sun, Oct 6, 2024. Continue.dev is another AI coding editor. It supports OpenRouter. So now I have heard good things about: #ai-coding-tools #code-agents #dev #github #prompt-engineering
  • Github Copilot
  • Cursor
  • Cody
  • Continue.dev (supports OpenRouter)
  • Aider (supports OpenRouter)
  • Maybe:
    • Codeium
  • Not:
    • Amazon Q Developer

• Sun, Oct 6, 2024. Parables on the Power of Planning in AI: Giving models about 30 seconds of thinking time consistently improves results - as much as increasing parameter size by a factor of 1,000 to 100,000! #best-practices #optimization #prompt-engineering
  • This works particularly well for verifiable results (code, math, etc.)
  • Technique: Ask an LLM hundreds of times at low temperature and pick the most common one. (Google's Minerva used this on the MATH dataset.)
  • Better Technique: Ask an LLM hundreds of times. Pick the best solution based on an evaluation metric (reward model)
  • Better Technique: Apply a reward model at EACH step of the process. OpenAI's "Let's Verify Step by Step"

• Fri, Oct 4, 2024. Software 2.0 by Andrej Karpathy. #ai-coding-tools #automation #dev #markdown #prompt-engineering
  • This is fundamentally altering the programming paradigm by which we iterate on our software, as the teams split in two:
    • the 2.0 programmers (data labelers) edit and grow the datasets, while
    • a few 1.0 programmers maintain and iterate on the surrounding training code infrastructure, analytics, visualizations and labeling interfaces.
  • Adaptive UI ideas:
    • Adaptive Fields: Show only required fields based on what the user field so far.
    • Smart Inputs: Dropdowns and auto-complete based on user's context.
    • Smart Themes: Change font size, contrast, theme guessing the user's age and preferences.
    • Dynamic Menus: Show what they might need to do next. Like Nokia's right button, but using LLMs.
    • Smart Tooltips: Check what the user's doing (delays, confusions, previous clicks, current actions) and show relevant tips.
    • Personalized Layout: Show only the relevant sections of the app. E.g. based on what they're doing.
    • Smart Charts: Create the right chart that solve the user's question.
  • Adaptive Back-end
    • Dynamic APIs: Create endpoints on the fly based on user needs
    • Dynamic Indexing: Create & update indices on the fly based on user needs
    • Dynamic Schema: Create & update schema on the fly based on user needs
    • Dynamic Migration: Migrate to a new database or OS or language as required
    • Dynamic Queries: Create SQL/NoSQL queries to solve the user problem
    • Dynamic RBAC: Figure out who needs permissions and why. Add OR REMOVE access as required
    • Dynamic Logging. Log what's required. Explain why it's logged and what's happening. Fix code that raised the error
    • Dynamic Caching. Cache what's likely to be required. Evict what may not be required. Figure out cache keys.

• Sat, Sep 28, 2024. API coding best practices Source via Simon Willison: #ai-coding-tools #best-practices #github #prompt-engineering
  • Always add screenshots to the Readme. They never break.
  • Always add every example. Human think in examples.
  • Avoid defaults and be explicit unless 99% of the usage is with the default.
  • Make the feedback loops incredibly fast.
  • Make deprecations easy for users to deal with.
  • Keep objects immutable.

• Sun, Sep 22, 2024. Sentient lets you control the browser via Python in natural language #prompt-engineering #python

• Fri, Sep 20, 2024. Prompt engineering for code generators: #ai-coding-tools #code-agents #dev #future #markdown #prompt-engineering
  • Claude Artifacts Prompt
  • Val.Townie system prompt. Good example of how to create
  • Cursor editing
  • Cursor debugging
  • Cursor conversation

• Fri, Sep 20, 2024. XML tags seem best to structure prompts across LLMs. #ai-coding-tools #code-agents #llm-ops #markdown #prompt-engineering
  • Claude
  • OpenAI
  • Gemini

• Fri, Sep 20, 2024. <link rel="modulepreload"> lets you load and compile modules early! #future #html #prompt-engineering

• Sun, Sep 15, 2024. Prompt engineering tip: Tell LLMs another AI wrote code. Else they will agree with you! #ai-coding-tools #automation #code-agents #future #llm-ops #prompt-engineering

• Tue, Sep 10, 2024. Simon Willison's crowdsourced list of prompt engineering hacks #future #prompt-engineering

• Tue, Aug 27, 2024. Cursor uses speculative edits and a variety of other techniques to speed up code editing. #ai-coding-tools #future #markdown #prompt-engineering

• Mon, Aug 19, 2024. Copilot's system prompt calls a search_enterprise(query: str) tool and a hint(M365Copilot_language: str) tool as assistants. #code-agents #prompt-engineering

• Mon, Aug 19, 2024. Anthropic Prompt Caching is 90% cheaper to use and 25% costlier to create. So if there's a 27% chance it'll be re-used, cache it. #optimization #prompt-engineering

• Mon, Aug 12, 2024. Code agent frameworks to explore: #ai-coding-tools #automation #code-agents #dev #future #github #markdown #prompt-engineering
  • Cognition
  • Factory
  • Codegen

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #TODO #embeddings #future #gpu #markdown #models #optimization #prompt-engineering

• Wed, Aug 7, 2024. Notes from a 4-hour flight: #ai-coding-tools #llm-ops #prompt-engineering
  • What We’ve Learned From A Year of Building with LLMs
    • Strategy
      • IS IT TOO HARD/EXPENSIVE? Log it. LLMs are getting cheaper and better.
      • WILL OPENAI BUILD IT? If so, wait for it instead of building.
      • HAS A STARTUP BUILT IT? If so, use it instead. It's a generic use case there's no point re-inventing.
      • FOCUSED USE CASES over generic. Build trust by starting small.
      • Tools for LLM Ops (feedback): LangSmith, Log10, LangFuse, W&B Weave, HoneyHive #TRY
      • Human in the Loop is about humans evaluating model outputs. That's different from AI in the loop, human in the center, where AI accelerates human output (like Github Copilot)
    • Operations
      • CHECK EMBEDDINGS DRIFT over time. Users might be input-ing different things than before.
      • LOG AND REVIEW everything.
      • Instructor coaxes structured output from LLM APIs. #TRY
      • IMPLICIT FEEDBACK collection is easy. Just let users edit stuff. #TRY
    • Tactical
      • Try n-shot prompting (n=5-12) before bigger models. #TRY
      • Always structure for output: Markdown, XML/HTML tags.
      • Combine RAG with Keyword search. It reduces user frustration in edge cases.
      • Prefer multiple small prompts to one big prompt. Do X. Then Y. Then Z.
      • Jitter prompts for diversity beyond temperature.
      • LLM-as-judge works better when comparing outputs (not rating 1 output). Keep length similar (LLMs prefer wordiness). Swap order and compare. Allow for ties. Ask for reason FIRST.
  • Hermes: A Text-to-SQL solution at Swiggy
    • "Hermes performed significantly better for charters with well-defined metadata and a relatively smaller number of tables."
    • "We collect feedback on the accuracy of the returned query from stakeholders directly within the Slack bot."
  • How I use AI and "Replacing my right hand with AI"
    • EMBED in every app/workflow. E.g. Auto-fix spellings. Auto-review code. Auto-ask LLM on errors and apply patch! Auto-search for answer, assess, continue.
    • PERSIST. Stick with the LLM to the end. Don't fix it yourself. It's faster. #TRY
      • INTERVENE FAST. If an LLM can't solve it by itself in 2 tries, it needs in-depth help.
    • APP-IFY one-off tasks. Disposable tools. "Write web-app to convert JSON to tab-delimited." "Extract fields as a table." "Diff JSON." #TRY
    • BEST language/frameworks preferred. CUDA in Python. Rust. C. Raspberry Pi. Arduino. Bluetooth. Modern ESM/JS. #TRY
    • TEACH examples. "Here's the LLM Foundry API." "Here's how to use gramex.data."
    • DUMP entire code. Models can handle it. Refactoring to SQLAlchemy 2, Pandas 2. API Documentation. Test case generation. #TRY
    • ASK for features & packages. Docker without root access. GPU access inside docker. Windows CLI-only C++ compiler.
    • TEST CASE writing. #TRY
    • SPEC IN DETAIL. Use these libraries. Write like this: code example.
    • SPEC USAGE in detail.
      • "I will just pipe it into sqlite", or "I will just run ffmpeg -i filename [YOUR OPTIONS].
      • Describe the UI, API input/output, data structure, and internal data structure.
    • HELP on usage. "ffmpeg to get audio.mp3".
  • My benchmark for large language models
    • LLM(text) is a useful function to have in JS and Python too. Useful as a simple pip install llmfoundry
    • Allow images, files in LLM()
  • Current list of #IMPOSSIBLE (or hard) things for LLMs
    • Translate technical documents to Dutch -- because they don't understand the technical terms well
    • Translate large documents (JSON to XML, English to Chinese, Python to Rust, Wrong to right spelling) -- because the output tokens are limited

• Sun, Jul 14, 2024. Github Copilot internals walks through how Copilot constructs its prompts #code-agents #future #prompt-engineering

• Tue, Jun 25, 2024. LLM Patterns include Evals, RAG, Fine-tuning, Caching, Guardrails, Defensive UX, Collect feedback. Notably: #ai-coding-tools #automation #best-practices #future #llm-ops #optimization #prompt-engineering
  • Defensive UX: Microsoft, Google, and Apple have guidelines for Human-AI interactions
  • Collect feedback: Explicit and implicit

• Thu, Apr 18, 2024. Effort engine introduces "effort" as a parametrizable way to speed up LLMs with a quality trade-off. Works on Mistral for now. #gpu #llm-ops #optimization #prompt-engineering

• Sat, Mar 30, 2024. browser.new is a good example of a browser agent. It slowly but independently does a good job of achieving the result. Example: What crew is common in Ingrid Bergman - Cary Grant films? #code-agents #prompt-engineering

• Sat, Mar 30, 2024. TechEmpower web server benchmarks place Rust servers on top #hosting #prompt-engineering #server #web-dev

• Wed, Mar 27, 2024. Representational engineering lets you control LLM output based on preference on the fly. #future #llm-ops #markdown #models #prompt-engineering

• Tue, Mar 19, 2024. Ways to use inversion #best-practices #dev #future #prompt-engineering
  • "Pre-mortem" is an analysis at the beginning of how a project failed. Then avoid that
  • "Red team" or "Black hat" are designated to contradict.

• Tue, Mar 19, 2024. Control vectors in real-time lets you control response in real-time #automation #future #prompt-engineering

• Sun, Mar 3, 2024. Making a tool more usable, e.g. a video, can have a 10-100X impact. Yet every developer thinks it's redundant #ai-coding-tools #prompt-engineering #web-dev

• Sat, Mar 2, 2024. You can use slots to stream HTML out of order! #future #html #prompt-engineering #web-dev

• Fri, Feb 23, 2024. Teknoturf is using Gen AI to #ai-coding-tools #automation #prompt-engineering #tts
  • Improve prompts when teaching prompt engineering.
  • Pronounce languages better, identifying which words Tamilians and Malayalis will mis-pronounce.

• Sat, Feb 17, 2024. Fine tuning makes economic sense only if the input tokens SAVED is twice the output token size on each call. #future #optimization #prompt-engineering

• Sat, Feb 17, 2024. Discussion with Karthik A #ai-coding-tools #github #prompt-engineering
  • #TODO Ask IIT students to do internship tasks. Use advent of code is a qualifying criterion
  • Tata motors unionized DB admins for longevity. No one can take their jobs. Hires people who LIKE their jobs
  • Rust gives me typing. It's very efficient. Pola.rs is interesting but Pandas as good enough.
  • Explore alerts from CCTV feeds. Karthik sends email alerts with pictures for:
    • "Is the machine on or off"? for productivity
    • "Are people not wearing helmets?" for safety at Cummins
  • #TODO Integrate with WhatsApp. Use LLMs with function calling for responses
  • Use expiring links (to pictures or content). It increases engagement
  • Check Deno licensing. Is there a commercial clause? #ANS No - it's MIT license
  • Centre or excellence for zero emission tech at IIT. Karthik is part of it
  • Explore auth0. 7000 users are free
  • toml is part of the Python 3.11 standard library!
  • If copilot writes code we don't understand we are screwed. Hence expertise matters

• Sun, Feb 11, 2024. Two pass generation is a clever technique to get multiple SEQUENTIAL answers in a single API request. For example the schema {'code', 'optimized_code'} will generate code and then optimize it. #ai-coding-tools #code-agents #future #optimization #prompt-engineering

• Tue, Jan 30, 2024. Hidden brain podcast. Making the most of your mistakes #prompt-engineering
  • FIX every small mistake. You never know how they might line up in the future
  • You also never know how small little things done well might line up to give you a boost in the future
  • The Toyota cord does not actually stop the production line. It brings a team lead over who quickly diagnoses the problem with you. The responsiveness of the league is a critical factor and so is encouragement
  • That isn't always a single bottleneck to stop that is the case of a simple failure. There can be a series of holes that happen to align perfectly.
  • These are events that lead to catastrophic failures or successes
  • Do as little as possible, waste as little as possible, until you know that the outcome is worthwhile.
  • Figure out what is the value of the outcome and the most important piece of information you need to discover that
  • Do full research before you try and fail. The aim of failure is learning at the least possible cost

• Sat, Jan 27, 2024. ⭐ OpenAI's prompt engineering strategies are an excellent start for prompt engineering. A few lessons: #automation #prompt-engineering
  • Use detailed system prompts, often containing the entire instruction set, if it won't change over the course of a conversation.
  • "... summary of the prior conversation could be included as part of the system message" is an interesting history compression tactic.

• Sat, Dec 16, 2023. Github Copilot's new features makes your entire workspace or a specific file its context. It also auto-writes your commit messages and PR descriptions. #github #markdown #prompt-engineering

• Thu, Dec 7, 2023. 12 Tuning Strategies for RAG #markdown #optimization #prompt-engineering

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding-tools #automation #code-agents #future #llm-ops #optimization #prompt-engineering

• Sat, Nov 25, 2023. Agents mimic programming primitives. Switch. While. Call a function. Print. #ai-coding-tools #automation #code-agents #prompt-engineering

• Sun, Nov 19, 2023. Each of the GPTs / Prompts we create could be like a UNIX command prompt, and become a startup of its own #chatgpt #future #prompt-engineering

• Thu, Oct 26, 2023. 26 Oct 2023: ChatGPT AutoExpert is a great prompt mechanism for higher accuracy, context and control. Example #ai-coding-tools #automation #chatgpt #code-agents #prompt-engineering #tts