Sun, Dec 29, 2024

• What would be the cost of storing about 500GB of LLM cache logs and 5 million write requests per month? #cloud
  • CloudFlare KV: $250 + $25 / month Ref
  • MongoDB: $125 + $5 / month Ref
  • S3: $0.0115 + $25 / month Ref + ?
  • CloudFlare R2: $0.0075 + $22.5 / month Ref

• Given about 30 generations, Llama 1b outperforms Llama 8b. Ref #llm-ops

• OpenAI introduced a developer role in addition to the system role. This is mainly for o1. The API is backward compatible - and also forward compatible. OpenAI #dev

• Em dashes are a strong sign of ChatGPT use. Curly quotes too. Reddit #chatgpt #markdown

• CloudFlare has multiple SSL modes when proxying requests.
  • Off (no encryption): No encryption between browsers and Cloudflare or between Cloudflare and origins. Everything is cleartext HTTP.
  • Flexible: Browsers to Cloudflare is HTTPS, Cloudflare to origin is HTTP. Useful to set up CloudFlare as a HTTP Proxy.
  • Full: Browser to Cloudflare matches browser request. Same protocol is used for Cloudflare to origin, without validating the origin’s certificate. Use for self-signed or otherwise invalid certificates.
  • Full (strict): Similar to Full Mode, but with validation.
  • Strict (SSL-Only Origin Pull): Cloudflare always connects to the origin over HTTPS with certificate validation.
  • Getting this wrong can lead to a HTTP 526: invalid SSL certificate

• aspose-words is a Python library that converts documents with many formats (Word, RTF, PDF, HTML, Markdown, EPUB, etc.) #document-conversion

• Discourse does not support searching across multiple forums. Instead, search for the term in all forums. Example. Then scroll through the results. Then, in the console, hide the ones you don't want. Example: #search
  • Hide posts that are not in the "Tools in Data Science" category: $(".badge-category__name").filter(d => d.textContent == "Tools in Data Science").map(d => d.closest(".fps-result")).filter(d => d).forEach(d => d.style.display = "none")

• A clever idea. Give an LLM a chapter from a textbook. Ask it to generate a unique, playable game to help me learn theconcepts for an exam. Page Bailey #llm-ops

• Satya Nadella prepares for meetings by asking Copilot to tell him everything he needs to know about the client from the CRM, emails, meeting transcripts etc. He shares that colleagues who annotate it further for him. That's using AI for reasoning and collaborating with colleagues. Satya Nadella | BG2 w/ Bill Gurley & Brad Gerstner #automation #prompt-engineering

• WOW. This is how a software agent will work alongside humans: Fix issue : Add color to the line next to "Ran a XXX Command" based on return value - using @openhands-agent. #automation #code-agents #future #5478

• aisuite by Andrew Ng is a unified interface to LLMs. Sort of like an openai library across multiple providers. #ai-coding #llm-ops

• Learnings from Best of 2024 in Agents (from on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands) #code-agents #dev #github #prompt-engineering #1
  • Passing code execution as a tool is more powerful than granular tools. You combine multiple tools and tool calls into one. You move code to the data rather than the other way around. Mostly, you need bash, Python (or Jupyter), file manager, web browser.
  • UI: Go where the user is, instead of bringing them to you.
  • A remote runtime is a critical component.
  • Claude 3.5 Sonnet (20241022) and Claude 3.5 Haiku (20241022) perform best on SWE Bench, followed by Deepseek V3, then O1 2024-12-17. X

• Since HNSW indexing is an overhead, just use NumPy matrix multiplication to calculate cosine similarity. For 1M vectors, it takes ~0.05 seconds. A 1M vector dataset handles ~2GB of text at a chunk size of 2K chars. In short, if you're embedding <2GB of text, just use NumPy.

• DuckDB's VSS extension HNSW index + Embeddings (2K chunks of 512 dimensions) takes up roughly 2.5X the size of the original data. Embedding 554 files of ~4,456 KB took 710 seconds. Creating the index took 660 seconds. The resulting DB was 18.1 MB. #embeddings

• How to use LLMs in market research. #llm-ops
  • Use LLMs with search for secondary research.
  • Create different personas and run user surveys on them. This paper used 1,052 real-life interview audio transcripts as agent memory to simulate people
  • Generate your market research report using LLMs.

• Medical coding is an area ripe for LLMs. #ai-coding #code-agents #llm-ops
  • Ojasvi Yadav created a repo that uses hierarchical classification (rather than embeddings) to find the right coding.
  • Gemini models seem to understand medical terms better than others.
  • RapidClaims, funded by TogetherAI, is apparently working on this problem.

• Document to Markdown Converters: #document-conversion #markdown
  • PyMuPDF4LLM uses MuPDF. Requires PyTorch.
    • PYTHONUTF8=1 uv run --with pymupdf4llm python -c 'import pymupdf4llm; h = open("pymupdf4llm.md", "w"); h.write(pymupdf4llm.to_markdown("$FILE.pdf"))'
  • markitdown from Microsoft. PDF via PDFMiner, DOCX via Mammoth, XLSX via Pandas, PPTX via Python-PPTD, ZIP, etc.
    • PYTHONUTF8=1 uvx markitdown $FILE.pdf > markitdown.md
  • Docling by IBM. Unable to install via pip on Windows AND on Linux.
  • MegaParse uses libreoffice, pandoc, tesseract-ocr, etc. Requires OpenAI API key.

• Awesome Tabular LLMs compiles encodings of tables for LLMs. #llm-ops

• What's the best way of encoding tabular data for LLMs? Looks like including the cell address helps. Here is an explanation from ChatGPT #llm-ops

• How are software engineers are future-proofing their careers in the face of LLMs? #llm-ops #prompt-engineering
  • Leveraging LLMs as Force Multipliers
    • Use LLMs for repetitive tasks, rapid prototyping, exploring multiple approaches, data extraction and brainstorming, providing feedback.
    • Explore prompting techniques, integrate LLMs into their workflows, and develop strategies for validating and refining LLM-generated code
  • Focusing on higher-level skills that llms struggle with
    • Systems Thinking and Architecture: code readability, extensibility, testability, and maintainability
    • Problem Solving and Critical Thinking: define problems clearly, break them down into manageable parts, and reason through complex scenarios. LLMs produce plausibly incorrect code.
    • Communication and Collaboration
    • Domain Expertise
  • Exploring Adjacent Roles: product management, technical leadership, or consulting. Involve more interaction with clients and stakeholders.
  • Developing "Evergreen" Skills: debugging, system administration, and security. Or outside of software engineering, such as trades or other hands-on vocations.
  • Scepticism: LLMs may not reach a level of sophistication that would render their expertise obsolete. Complex problems, understanding context, and producing high-quality, maintainable code.

• Examples of agentic AI #ai-coding #automation #code-agents #llm-ops #prompt-engineering
  • Text-to-SQL automated business analyst: A system that generates SQL queries from natural language, handles errors, creates visualizations, and includes a FAQ component. The author calls it "constrained agentic AI."
  • Data source querying system: A bot that queries multiple SQL and API data sources, selecting tools and reformulating tasks as needed.
  • Cursor (agentic mode): An LLM-powered VS Code fork that chains together various LLM capabilities (code generation, applying changes, linting suggestions, terminal commands, codebase RAG) to reduce user prompts.
  • Vulnerability finding system: A system that uses LLM agents to discover novel vulnerabilities in open-source web applications. The agents leave traces of their actions.
  • Marketing strategy generation system: A system using approximately 60 agents to generate marketing strategies.
  • Restaurant finder: A system that searches for restaurants based on dietary preferences and group size, and downloads social media information.
  • Proofreading and editing of transcripts: LLM agents apply specific customer requirements to transcripts after human editing.
  • Meeting notes and action items generator: A system that generates meeting notes and action items.
  • O'Reilly auto parts customer service agent: An agent demonstrated using RAG.
  • UI enhancement agent: An agent that added features like language locales and dark mode to a UI.