#markdown

• Sat, May 31, 2025. Planning to use pretty-quick instead of prettier. It's a wrapper that only fixes changed files based on git. #ai-coding-tools #future #github #markdown #prompt-engineering

• Tue, May 27, 2025. DuckLake is a DuckDB extension that makes Parquet files editable with history. And much more. DuckDB #database #markdown

• Tue, May 20, 2025. just is a command runner like make but uses YAML conifguration. Written in Rust. #github #markdown

• Fri, May 16, 2025. DeerFlow is an open source Deep Research MCP. Lets you run deep research outside of the standard chatbots. #markdown

• Tue, May 13, 2025. Today, if I had to store a bunch of data files (e.g. parquet) under 1GB, here are options: #cloud #database #github #markdown
  • GitHub Releases. 2 GiB per file, unlimited total & bandwidth. 🟢 Immortal URL, versioning, easy CI publish. 🔴 Each file must stay < 2 GiB; no built-in SQL.
  • Zenodo (CERN). 50 GB per record; one-off bumps to 200 GB. 🟢 DOI assignment, archival mandate. 🔴 Occasional throttled bandwidth; no API for partial file reads.
  • Hugging Face Hub. 300 GB per repo; 50 GB per file. 🟢 Git-based, dataset tooling, lively ML community. 🔴 Large files need git-LFS; pushes via LFS can be slow.
  • Cloudflare R2. 10 GB storage & 1 M ops / month. 🟢 S3 API, zero-egress to Cloudflare Workers, fast. 🔴 10 GB cap below your 50 GB target.
  • Kaggle Datasets. 20 GB per dataset, public only. 🟢 Built-in notebooks & GPU. 🔴 No programmatic SQL API; quotas sometimes change.
  • data.world (free). 1 GB total, 100 MB per dataset. 🟢 Nice social features. 🔴 Too small for your size.

• Mon, May 12, 2025. We could "fine-tune" system prompts automatically with evals, creating a "system prompt learning" paradim -- like my promptevals. Andrej Karpathy #ai-coding-tools #automation #future #markdown #optimization #prompt-engineering

• Sun, May 11, 2025. Pandoc has several options useful when converting Markdown to HTML (cat file.md | pandoc -f markdown -t html). My favorites: #document-conversion #html #markdown
  • --no-highlight skips code-highlighting. --highlight=pygments adds Pygments styling
  • --wrap=none doesn't wrap the content in a single block
  • --number-sections adds section numbering (<h2>1. Introduction</h2>)
  • --shift-heading-level-by=NUM – shift all headings by NUM levels (e.g., start at <h2> instead of <h1>)
  • pandoc -f markdown-auto_identifiers drops the auto-identifiers extension that generates id=... for each heading
  • pandoc -f gfm uses GitHub flavored Markdown. Run pandoc --list-extensions=gfm to identify the extensions it uses.
  • Pandoc's Markdown extension examples are quite extensive.
  • Auto-enabled GFM extensions:
    • alerts: GitHub-style callouts (info, tip, warning) via > [!TYPE] blocks.
    • autolink_bare_uris: Turns bare URLs into links, without needing <...>.
    • emoji: Parses :smile:-style codes into Unicode emoji characters.
    • footnotes: Enables footnote syntax with [^id] and definitions at the bottom.
    • gfm_auto_identifiers: Uses GitHub’s heading-ID algorithm: spaces → dashes, lowercase, removes punctuation.
    • pipe_tables: Enables table.
    • raw_html: Raw HTML is unchanged.
    • strikeout: Enables strikethrough with ~~text~~.
    • task_lists: Parses - [ ] and - [x] items as checkboxes.
    • yaml_metadata_block: YAML front matter for document metadata, e.g. <title>
  • GFM extensions worth enabling:
    • ascii_identifiers: Strips accents/non-Latin letters in automatically generated IDs.
    • bracketed_spans: [Warning]{.alert} becomes <span class="alert">
    • definition_lists: Term\n: Definition text becomes a definition list
    • fenced_divs: ::: {.note} block creates a <div class="note">...</div>
    • implicit_figures: Standalone images become <figure> with <figcaption>.
    • implicit_header_references: [Section] is treated as [Section][#section]
    • raw_attribute: <b>bold</b>{=html} is inserted as HTML
    • smart: Converts straight quotes to curly, -- to en-dash, --- to em-dash, ... to ellipsis.
    • subscript & superscript: E.g. H~2~O and E = mc^2^

• Sat, May 10, 2025. Sli.dev is a Markdown slide language. Similar to Marp #ai-coding-tools #github #html #markdown

• Wed, May 7, 2025. When writing notes, summarize at the end of the day the learnings and next steps. #future #markdown

• Mon, Apr 28, 2025. Streamdown is a CLI markdown streaming processor. uvx streamdown --exec 'llm chat' lets you chat with an LLM using Markdown formatting. It's still a little rough at the edges. #llm-ops #markdown

• Sun, Apr 27, 2025. With LLMs writing code, it's becoming practical to run so many more things in SQL -- such as parsing HTML. Simon Willison #ai-coding-tools #automation #code-agents #dev #future #html #llm-ops #markdown #prompt-engineering

• Thu, Apr 24, 2025. O4-Mini-High is great at going through an under-documented repo and finding things. For example, here's how I configured cmdg. #ai-coding-tools #code-agents #github #llm-ops #markdown #prompt-engineering

• Wed, Apr 23, 2025. semantic-text-splitter could be the go-to text splitter. It's Rust-based, supports MarkdownSplitter, and multiple tokenizers. Alternatives like semchunk, advanced-chunker, chonkie, etc. seem clunkier. #ai-coding-tools #markdown

• Tue, Apr 22, 2025. You can run xclip -sel clip -o | pandoc -f markdown -t html --no-highlight | xclip -sel clip -t text/html -i to convert Markdown in the clipboard to rich text. But xclip doesn't support multiple selections, so the text is lost. ChatGPT #document-conversion #markdown

• Tue, Apr 22, 2025. DuckDB UI & Notebooks will potentially be a good alternative to Datasette, DBeaver, etc. But for now, there are still glitches. It crashes with a SIGSEGV (Address boundary error) when connecting to SQLite databases. #dev #database #markdown

• Mon, Apr 21, 2025. DuckDB's CSV parser might be one of the most forgiving parsers. Even better than Pandas or SQLite3. Ref #database #markdown

• Sun, Apr 20, 2025. Ink is like React for the CLI. #github #markdown

• Sun, Apr 20, 2025. Using Kiran's macOS setup for dev I enabled colorized less and mouse options for tmux. #ai-coding-tools #github #markdown

• Sun, Apr 20, 2025. Augment Code is an AI code editor that's growing popular on Reddit. #ai-coding-tools #code-agents #future #github #markdown

• Sat, Apr 19, 2025. The devcontainers.json spec encapsulates everything you need to get a codebase running for development - as opposed to production. E.g. VS Code extensions, linters, etc. #dev #markdown

• Sat, Apr 19, 2025. AI coding can teach us new tech. For example I learned that tqdm.pbar can print logs while showing progress. It's worth noting such learnings until it becomes a habit. #ai-coding-tools #automation #code-agents #future #github #markdown #prompt-engineering

• Sat, Apr 19, 2025. If English is the new coding language, should prompts be versioned? Or at least stored, perhaps in a PROMPTS.md? #ai-coding-tools #code-agents #future #markdown #prompt-engineering

• Sat, Apr 19, 2025. marimo new "prompt" generates an entire new notebook using your prompt. Video #markdown

• Fri, Apr 18, 2025. Codex is more a proof-of-concept for agentic coding than a coding tool. #ai-coding-tools #code-agents #github #markdown
  • You can't run commands. Only prompts. You need to exit codex to run commands. So you can't use it like a shell, e.g. like Warp.dev.
  • It doesn't index local code. It runs commands to figure out stuff.
  • Code diffs and applying changes are clunky.
  • The output is hard to read with text scrolling.
  • codex.md can only handle 32K.

• Sun, Apr 13, 2025. Marp supports bespoke transitions which includes morphing animations. This can create a bar chart race just using Markdown! #markdown

• Thu, Apr 10, 2025. Claude 3.7 Sonnet with extended thinking has a token limit of over 64,000 tokens. Given a strong instruction following capability, that makes it one of the most powerful models for transforming text. For example, transcription restyling, translations, XML to json conversions, PDF to XML, etc. #document-conversion #markdown #speech-to-text

• Tue, Apr 8, 2025. One way to copy as Markdown: copy page contents, paste in text-html.com, copy HTML, paste in Turndown, copy Markdown. #document-conversion #html #markdown

• Tue, Apr 8, 2025. Claude 3.7 Sonnet with extended thinking has a token limit of over 64,000 tokens. Given a strong instruction following capability, that makes it one of the most powerful models for transforming text. For example, transcription restyling, translations, XML to json conversions, PDF to XML, etc. #document-conversion #markdown #speech-to-text

• Thu, Apr 3, 2025. Clipboard2Markdown is a utility that lets you paste rich text and convert it to Markdown. #document-conversion #html #markdown

• Wed, Apr 2, 2025. Learnings on AI code editors via Deep Research from ChatGPT, Gemini, Grok, Perplexity: #ai-coding-tools #code-agents #github #markdown #prompt-engineering
  • GitHub Copilot can identify the source of a code snippet as a repo. That helps with copyright issues.
  • Cursor uses a shadow workspace - a temporary sandbox where it edits files before applying changes at one shot.
  • Cursor auto-complete has context of other files, i.e. inserting an class in a .js file based on another HTML file's contents.
  • Windsurf seems to be best for large code bases and for large-scale refactoring. It can also run test results fix them.
  • Windsurf includes a browser and lets you click on an element and prompt to change its behavior, etc. That's good for front-end developers.
  • Roo Code can run scripts as part of the workflow, letting you run linting, tests, starting web apps, query databases, etc.
  • Roo Code lets you create persona, e.g. code reviewer, data storytelling and analysis, etc. with access to different tools and behaviors.
  • Roo Code does not support auto-complete.

• Sat, Mar 29, 2025. Discussion with Vedang #ai-coding-tools #code-agents #dev #github #llm-ops #markdown #prompt-engineering
  • Recurse center (Brooklyn, online) is a 6/12 week free self-driven programmer retreat. Runs every 6 weeks. You can do whatever you pick. There are daily standups for accountability. The groups are diverse. You can pair with them, pivot ideas, whatever. Principles: push yourself & learn.
  • Western education techniques (e.g. spaced repetition, adaptive learning) are very much present in Indian coaching systems, though not known by those names. However, interventions are hard since class 12 students just don't have enough time.
  • Coaching classes are a social phenomenon. It's not the smart students who are pulling in their friends. Smart students actually follow the popular students. (Coaching classes are below the typical smart students' standards.)
  • Monetizing coaching is hard. People don't want to pay for advice, and welcome free advice only if they ask for it. Coupling with execution is necessary.
  • Aider's integrations make it more powerful than Cursor/Windsurf. It auto-lints, runs test cases. Allows different models for "architecting" (generating changes) vs "editing" (applying code). It reads from the screen logs. Context is manual, not automated. Uses an ai! comment to trigger changes and ai? to ask questions.
  • Cline.bot is another Cursor-like open source AI code editor that's a VS Code plugin.
  • When coding with LLMs, a useful workflow is: data schema ➡️ interfaces ➡️ LLM-generated test cases ➡️ code.
  • ShellSage is a tmux based LLM tool for the command line. It screen-grabs from tmux, which is powerful.
  • Some MCPs that have proven useful: vega-lite, SQLite, sequential thinking, memory
  • make sucks but is hard to beat. just comes closest.
  • CRDTs are more powerful than for just collaborative editing. It can power a peer-to-peer Internet (beginning with office tools). Versioning schema is still problematic. yjs is a good start but automerge (Rust, WASM) is faster and may be better.
  • Fermyon hosts WASM serverless functions.

• Fri, Mar 28, 2025. There's an actual convention for using emojis in Git commits: gitemoji. It even has a VS Code plugin, a changelog generator, and more. #ai-coding-tools #github #markdown

• Fri, Mar 28, 2025. Emojis have a strong role in enhancing Markdown documents. The ones I use often are: #markdown
  • 🔴🟡🟢 for low/medium/high priority
  • ⭐️ or ❤️ or 👍 for ratings or emphasis
  • ✅ for completed tasks
  • 💡 for ideas
  • ⚠️ or ❗️ for warnings / issues

• Tue, Mar 25, 2025. Cursor with Claude 3.7 Max seems surprisingly good at generating multi-page sites at one shot. Potentially, it can edit large repositories of code as well at one shot. If that's the case, the way we write code will require higher order thinking skills: broad sweeping changes rather than micro edits. #ai-coding-tools #code-agents #dev #github #html #markdown #prompt-engineering #web-dev

• Thu, Mar 20, 2025. Andrej Karpathy's note taking mechanism is similar to mine, except I use Microsoft TODO. Ref #github #markdown
  • I have 3 categories. Things I learnt, which I just note. Things to explore, which I can delegate, defer, drop, or do at any time. Things to do, which are the hardest and pile up.

• Tue, Mar 18, 2025. Block notes is a notion like library for editor components. Converts to Markdown #html #markdown

• Sun, Mar 16, 2025. duckdb -ui launches a DuckDB notebook. This is built into newer DuckDB releases #markdown

• Sat, Mar 15, 2025. htmlq and pup query HTML. They're like jq for HTML. #github #html #markdown #prompt-engineering #web-dev

• Sat, Mar 15, 2025. LibreOffice is now on WASM. ZetaJS provides office in the browser. Has a CDN (that was down from our IP). 35M packaged binary. 100M of in-memory file-system loaded. #document-conversion #html #markdown #web-dev
  • Useful for: Document conversion, Thumbnail generation, Text extraction, Merging / splitting documents

• Fri, Mar 14, 2025. Twine (docs) is an open source interactive fiction / story writing tool. #ai-coding-tools #github #markdown #tts
  • Snowman is a browser-based Twine 2 story template format.
  • These enable behavioural experimentation. Cheaper than using tools like Gorilla.sc and Pavlovia for behavioral experiments
  • For example, you can present a social or political issue and see if people change their opinions more or less depending on the content/path they see. Or, if it varies by demographics. Or, check if repeated mentions or emotional hooks improve memory / retention. More research ideas

• Fri, Mar 14, 2025. Unsloth might be one of the simplest ways of fine-tuning. #automation #markdown #optimization #prompt-engineering

• Thu, Mar 13, 2025. Marp is my new favorite way to generate slides from Markdown. Reveal.js is not easy with Markdown (though HTML works well.) #ai-coding-tools #github #markdown
  • The VS Code plugin makes development very easy
  • Marp CLI makes deployment easy.
  • I used it for my talk on LLM Hallucinations (source).
  • Supports all bespoke features and plugins
  • Transitions. Requires OS animation effects to be enabled
  • Animated SVG backgrounds are a good add-on.

• Tue, Mar 11, 2025. Bots need structured content (e.g. Markdown, XML). Humans need rich content (e.g. HTML). Here are 4 ways to serve both, roughly in increasing order of sophistication: #code-agents #html #markdown #prompt-engineering
  1. Different URLs. E.g. https://example.org/about/ vs https://example.org/about.md (this is how Jekyll or Hugo work). Use for static sites generators.
  2. JavaScript. Inject after Markdown: <script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script><script>document.body.innerHTML = marked(document.body.textContent);</script>. Use for dynamically generated static sites.
  3. URL query parameters. E.g. ?format=markdown vs ?format=html vs ?format=json. Use in APIs.
  4. Content Negotiation. Based on the user agent and Accept header, serve Markdown or HTML. Send Vary: Accept to indicate that the response depends on the Accept header. Use for dynamic web apps.

• Sat, Mar 8, 2025. The Zettelkasten note-taking method proposes that you: #markdown
  • Capture: Write down every idea or piece of information on a separate note. Use your own words to ensure understanding.
  • Organize: Consolidate fleeting notes into permanent ones. Assign unique identifiers to each note for easy reference.
  • Connect: Link related notes to form a web of knowledge. This can be done with tags, references, or hyperlinks in digital systems.
  • Review: Regularly revisit your notes to strengthen connections and discover new insights.

• Sat, Mar 8, 2025. "Export to prompt" can be a useful feature in apps (or even as a bookmarklet). It would let you export content in an LLM-friendly Markdown format. You can paste it into an LLM and ask questions. Here are things I would find useful: #document-conversion #github #markdown #prompt-engineering
  • Copy an entire issue (with history) from GitHub, Gitlab, or JIRA
  • Copy an entire PR (with code changes) from GitHub, Gitlab, or Bitbucket
  • Copy CI/CD logs from GitHub Actions, Gitlab CI, Azure DevOps, etc.
  • Copy entire conversation thread in Gmail or Discourse, Service now etc.
  • Copy product reviews from Amazon, Shopify, etc.
  • Copy page(s) from wikis and content sites like Wikipedia, StackOverflow, etc.
  • Copy survey responses from Google Forms, Typeform, etc.
  • Copy all interactions with a contact (including interactions, proposal history) from HubSpot or Salesforce
  • Copy transcripts from Zoom, Teams, Google Meet, etc.
  • Copy as Markdown from Word, GDocs, PDF or HTML
  • Copy the summary of an analysis as well as all key metrics from any dashboard
  • Copy SAP invoices
  • Copy JDs, CVs, and reviews from Workday, BambooHR, DarwinBox, etc.
  • Copy design specs, component libraries, and style guides from Figma, Miro, etc.
  • Generated with the help of ChatGPT -- link not working

• Fri, Mar 7, 2025. Mistral released an impressive OCR model. #ai-coding-tools #document-conversion #github #gpu #markdown
  • Marker from DataLab seems comparable but is CC-BY-NC-SA.
  • MinerU convert medical textbooks to Markdown well.
  • Gemini Flash may be more cost effective and better

• Sun, Feb 23, 2025. VS Code can be used as a code executor by adding {"key": "shift+enter", "command": "workbench.action.terminal.runSelectedText", "when": "editorFocus"} to the keybindings.json file. Press Shift-Enter to run the selection on the terminal. Useful for DuckDB, SQLite, etc. Ref #ai-coding-tools #dev #markdown

• Thu, Feb 20, 2025. Mermaid has an Architecture Diagrams Syntax (in beta) that's capable of creating elegant architecture diagrams with icons. #markdown

• Thu, Feb 13, 2025. I switched to fish shell mainly because: #ai-coding-tools #github #html #markdown
  • Autocomplete and tab completion works perfectly, out-of-box.
  • Syntax highlighting is beautiful
  • Great multi-line editing

• Mon, Feb 3, 2025. GitHub models is free for anyone to try. The model catalog us extensive and even includes o3-mini which was launched this week (though in limited preview). #future #github #markdown #models

• Sun, Jan 12, 2025. GitHub CodeSpaces seems to be coming up more often in my radar, but I'm yet to figure out a use for it. #ai-coding-tools #cloud #future #github #markdown

• Sat, Jan 11, 2025. Measuring developer productivity with the DX Core 4 is a framework for measuring developer productivity. It encapsulates other frameworks like DORA, SPACE, and DevEx. #ai-coding-tools #best-practices #dev #markdown

• Sat, Jan 11, 2025. Can LLMs write better code if you keep asking them to “write better code? A delightful exploration of how Claude 3.5 Sonnet keeps optimizing and adding features to improve code. My takeaway: repeatedly applying a prompt gives us interesting new directions to explore. #ai-coding-tools #code-agents #llm-ops #markdown #optimization #prompt-engineering

• Fri, Jan 3, 2025. Redash and Metabase are new open source data visualization tools sitting alongside Grafana and Apache Superset. #markdown
  • Redash feels too clunky / enterprise-y rather than open-source-y.

• Thu, Jan 2, 2025. Tools that convert files to prompt / Markdown suitable for LLMs: #ai-coding-tools #document-conversion #github #markdown
  • uvx files-to-prompt
  • npx git-ingest
  • ingest - written in Go, only Mac/Linux binaries

• Mon, Dec 23, 2024. Em dashes are a strong sign of ChatGPT use. Curly quotes too. Reddit #chatgpt #markdown

• Mon, Dec 23, 2024. Document to Markdown Converters: #document-conversion #markdown
  • PyMuPDF4LLM uses MuPDF. Requires PyTorch.
    • PYTHONUTF8=1 uv run --with pymupdf4llm python -c 'import pymupdf4llm; h = open("pymupdf4llm.md", "w"); h.write(pymupdf4llm.to_markdown("$FILE.pdf"))'
  • markitdown from Microsoft. PDF via PDFMiner, DOCX via Mammoth, XLSX via Pandas, PPTX via Python-PPTD, ZIP, etc.
    • PYTHONUTF8=1 uvx markitdown $FILE.pdf > markitdown.md
  • Docling by IBM. Unable to install via pip on Windows AND on Linux.
  • MegaParse uses libreoffice, pandoc, tesseract-ocr, etc. Requires OpenAI API key.

• Mon, Dec 16, 2024. markdown2 is the new de facto Markdown library for Python. #markdown

• Sat, Dec 14, 2024. **/*.md can search for all Markdown files. Julia Evans #markdown

• Sat, Dec 14, 2024. Roaming RAG is an alternative to RAG without the vector database. #markdown
  • Applicable to well structured documents, e.g. technical books, manuals, etc.
  • Create a hierarchical outline of the document. Code
    • Keep the top-level headings.
    • Preserve the first ~100 characters of opening text from each section.
    • Present the second-level headings, but without any subsidiary content.
    • Provide each section a unique 8 digit hex identifier.
    • Each section heading is followed by a guiding comment for the model: Section collapsed - expand with expand_section("{identifier}").
  • Then read the relevant sections as context to answer the question. Code

• Wed, Nov 27, 2024. It's hard to summarize a large document using RAG. But you can directly add answers to such questions into the corpus, e.g. by adding a "summary" section, and other answers to common questions. #markdown

• Tue, Nov 26, 2024. CloudFlare workers can bundle any kind of files, including text, data, and WASM. Docs #cloud #document-conversion #markdown

• Sat, Nov 23, 2024. A list of Markdown to Website converters on this thread: #document-conversion #github #html #markdown
  • Jekyll - Ruby - 2008
  • MkDocs - Python - 2014
  • GitBook - JavaScript (Node.js) - 2014
  • MkDocs Material - Python (MkDocs-based) - 2016
  • Docsify - JavaScript - 2016
  • MdBook - Rust - 2017
  • Antora - JavaScript (Node.js) - 2017
  • Docusaurus - JavaScript (React) - 2017
  • JupyterBook - Python - 2019
  • Keenwrite - Java - ~2019
  • Honkit - JavaScript (GitBook fork) - 2019
  • Nextra - JavaScript (Next.js) - 2020
  • Astro - JavaScript/TypeScript - 2021
  • Hugo Book - Go (Hugo-based) - ~2020
  • Clowncar - JavaScript/Node.js - ~2021
  • Quarto - R and Python - 2022
  • Starlight - JavaScript/TypeScript - 2023

• Fri, Nov 22, 2024. DuckDB has an LLMs.txt. #database #github #llm-ops #markdown Today, 38 repos on GitHub support it

• Fri, Nov 22, 2024. Style of writing impacts output style a lot. E.g. Adding an evil laugh makes Claude more creative. Ethan Mollick #markdown

• Fri, Nov 22, 2024. For good structured mode output, we need good prompting. #ai-coding-tools #code-agents #markdown #prompt-engineering
  • Mentioning examples and schema and "JSON" helps. When providing examples, using (user, assistant) message pairs helps (I think it's because it's easier for the LLM to parse).
  • Using a {reasoning, answer} schema (with reasoning first) helps. Make reasoning concise and relevant Ref Arxiv
  • We already know code in JSON is not a great idea. Ref

• Mon, Nov 18, 2024. You can enable syntax highlighting just using fonts. Ref #ai-coding-tools #github #html #markdown

• Mon, Nov 18, 2024. HTML is all you need shows examples of using HTML for notebooks instead of Jupyter, Observable, etc. #html #markdown #web-dev

• Mon, Nov 11, 2024. Pass a complex data structure to Claude.ai and have it create an app to visualize it. It does well. Simin Willison #automation #markdown

• Fri, Nov 8, 2024. GraphRAG is better if data is naturally graph-structured. Else, it's slow and fills up the context window with even vaguely related stuff. Vigneshbabu, AMAT. #gpu #markdown

• Fri, Nov 8, 2024. uithub converts GitHub repos to Markdown. Just replace "g" in "github.com/..." with "u". Example #github #markdown

• Wed, Nov 6, 2024. Docling by IBM converts PDF, DOCX, etc. to Markdown. Like PyMuPDF4LLM but better. #document-conversion #markdown

• Mon, Nov 4, 2024. Zed.dev is an AI code editor by the creators of Atom. It's written in Rust and is blazing fast. It has native AI integration. #ai-coding-tools #code-agents #dev #github #markdown

• Thu, Oct 31, 2024. /llms.txt is a proposal to standardize /llms.txt files as a way to share LLM prompts. #llm-ops #markdown
  • Jina AI Meta Prompt is an example
  • Remotion system prompt is an example
  • https://docs.fastht.ml/llms-ctx.txt
  • https://docs.fastht.ml/llms-ctx-full.txt

• Wed, Oct 30, 2024. structuredClone deep clones objects in JS #markdown

• Sun, Oct 27, 2024. AI code editors are not good with large code bases today. Keep the refactoring exercises to below 1000 lines. Also evaluate the ease of setting it up locally #ai-coding-tools #code-agents #github #markdown

• Fri, Oct 11, 2024. DuckDB lets you create functions = macros #future #markdown

• Sun, Oct 6, 2024. Marimo supports: #markdown
  • Publishing any notebook to static.marimo.app as a static app
  • Creating a SINGLE link that embeds the ENTIRE notebook in the URL!
  • Runnable via uvx marimo edit

• Sun, Oct 6, 2024. Late chunking is an interesting approach to adding context to embeddings. (I don't understand it, but it's cheap and effective.) #embeddings #markdown

• Fri, Oct 4, 2024. Software 2.0 by Andrej Karpathy. #ai-coding-tools #automation #dev #markdown #prompt-engineering
  • This is fundamentally altering the programming paradigm by which we iterate on our software, as the teams split in two:
    • the 2.0 programmers (data labelers) edit and grow the datasets, while
    • a few 1.0 programmers maintain and iterate on the surrounding training code infrastructure, analytics, visualizations and labeling interfaces.
  • Adaptive UI ideas:
    • Adaptive Fields: Show only required fields based on what the user field so far.
    • Smart Inputs: Dropdowns and auto-complete based on user's context.
    • Smart Themes: Change font size, contrast, theme guessing the user's age and preferences.
    • Dynamic Menus: Show what they might need to do next. Like Nokia's right button, but using LLMs.
    • Smart Tooltips: Check what the user's doing (delays, confusions, previous clicks, current actions) and show relevant tips.
    • Personalized Layout: Show only the relevant sections of the app. E.g. based on what they're doing.
    • Smart Charts: Create the right chart that solve the user's question.
  • Adaptive Back-end
    • Dynamic APIs: Create endpoints on the fly based on user needs
    • Dynamic Indexing: Create & update indices on the fly based on user needs
    • Dynamic Schema: Create & update schema on the fly based on user needs
    • Dynamic Migration: Migrate to a new database or OS or language as required
    • Dynamic Queries: Create SQL/NoSQL queries to solve the user problem
    • Dynamic RBAC: Figure out who needs permissions and why. Add OR REMOVE access as required
    • Dynamic Logging. Log what's required. Explain why it's logged and what's happening. Fix code that raised the error
    • Dynamic Caching. Cache what's likely to be required. Evict what may not be required. Figure out cache keys.

• Mon, Sep 30, 2024. Coders are more likely to edit LLM code. Non-coders don't have that bad habit. #ai-coding-tools #code-agents #dev #github #llm-ops #markdown
  • Vaishnavi and Ranjeet edited code
  • Indal and Koustav didn't

• Sat, Sep 28, 2024. PyMuPDF4LLM can convert PDFs to Markdown. It handles tables, too. #document-conversion #llm-ops #markdown
  • 04 Oct 2024. PDF-Extract-Kit does PDF layout, formula, table, and OCR extraction using various models.
  • 04 Oct 2024. llmsherpa extracts PDF layout, tables, not OCR

• Fri, Sep 20, 2024. Prompt engineering for code generators: #ai-coding-tools #code-agents #dev #future #markdown #prompt-engineering
  • Claude Artifacts Prompt
  • Val.Townie system prompt. Good example of how to create
  • Cursor editing
  • Cursor debugging
  • Cursor conversation

• Fri, Sep 20, 2024. XML tags seem best to structure prompts across LLMs. #ai-coding-tools #code-agents #llm-ops #markdown #prompt-engineering
  • Claude
  • OpenAI
  • Gemini

• Tue, Aug 27, 2024. Cursor uses speculative edits and a variety of other techniques to speed up code editing. #ai-coding-tools #future #markdown #prompt-engineering

• Mon, Aug 19, 2024. LLMs write worse code in JSON than Markdown #ai-coding-tools #llm-ops #markdown

• Mon, Aug 12, 2024. Code agent frameworks to explore: #ai-coding-tools #automation #code-agents #dev #future #github #markdown #prompt-engineering
  • Cognition
  • Factory
  • Codegen

• Sun, Aug 11, 2024. DocxTemplater is SlideSense but open-core and handles DOCX as well! #document-conversion #html #markdown

• Sat, Aug 10, 2024. Embedding models can be fine-tuned. Example: #TODO #embeddings #future #gpu #markdown #models #optimization #prompt-engineering

• Wed, Jul 24, 2024. If LLMs can generate any text, and text can describe the real world, we can rapidly generate "artifacts" that generate: #markdown
  • 3D Printable Models:
    • STL (Stereolithography): Defines the surface geometry of 3D objects using triangular facets.
    • OBJ (Wavefront OBJ): Describes 3D geometry including vertices, textures, and normals.
    • X3D: An XML-based file format for representing 3D computer graphics.
  • Vector Graphics:
    • SVG (Scalable Vector Graphics): Defines vector-based graphics in XML format, useful for illustrations, diagrams, and user interface elements.
  • CAD Drawings:
    • DXF (Drawing Exchange Format): Represents CAD data, including shapes, lines, and curves, used in engineering and architecture.
  • Circuit Designs:
    • KiCAD: An open-source software suite for Electronic Design Automation (EDA), which uses various file formats like PCBNew and EESchema to represent circuit designs.
  • Blueprints and Architectural Designs:
    • GML (Geography Markup Language): Encodes geographical features and spatial information.
    • CityGML: A specific GML application schema for modeling and exchanging 3D city models.
  • Molecular Structures:
    • PDB (Protein Data Bank): Describes the three-dimensional structures of molecules.
    • CML (Chemical Markup Language): An XML-based standard for representing molecular data.
  • Robotics and Automation:
    • URDF (Unified Robot Description Format): Defines the physical configuration of a robot, including joints, links, and sensors.
    • COLLADA (Collaborative Design Activity): An XML-based schema to describe digital assets for 3D applications, often used in robotics.
  • Geospatial Data:
    • KML (Keyhole Markup Language): Used for geographic data visualization, primarily in Google Earth.
    • GeoJSON: A format for encoding a variety of geographic data structures using JSON.
  • Mathematical Markup:
    • MathML (Mathematical Markup Language): Describes mathematical notation and captures both its structure and content.
  • Music and Sound:
    • MusicXML: Encodes sheet music in a structured format that can be easily shared between different music notation software.
  • Documents and Text:
    • DocBook: A semantic markup language for technical documentation.
    • Markdown: A lightweight markup language with plain text formatting syntax.
  • Biological Data:
    • SBML (Systems Biology Markup Language): Represents computational models of biological processes.
    • PhyloXML: An XML format for representing phylogenetic trees.
  • Game Development:
    • FBX (Filmbox): A file format for 3D animation that can hold information about the geometry, textures, and animations.
    • VRML (Virtual Reality Modeling Language): Describes interactive 3D objects and worlds.
  • Data Visualization:
    • ChartML: Encodes charts and graphs in a structured format.
    • D3.js (Data-Driven Documents): Uses HTML, SVG, and CSS to bring data to life with interactive visualizations.
  • Building Information Modeling (BIM):
    • IFC (Industry Foundation Classes): Describes building and construction data.
  • Textiles and Fabrics:
    • LoomML: Represents the design and structure of woven fabrics.
  • Augmented Reality and Virtual Reality:
    • ARML (Augmented Reality Markup Language): Defines how augmented reality applications should behave and what content they should display.
    • VRML (Virtual Reality Modeling Language): For describing interactive 3D objects and worlds.
  • Medical Imaging and Health Data:
    • DICOM (Digital Imaging and Communications in Medicine): Encodes medical imaging data.
    • HL7 (Health Level 7): A set of standards for the exchange of information between medical applications.
  • Simulation Data:
    • FMI (Functional Mock-up Interface): Represents and exchanges dynamic simulation models.
    • SBML (Systems Biology Markup Language): For computational models of biological processes.
  • Sound and Audio:
    • MML (Music Markup Language): For encoding music notation and performance information.
    • SoundFont: A file format for defining musical instrument sounds.
  • Animation and Visual Effects:
    • BVH (Biovision Hierarchy): Encodes motion capture data.
    • Alembic: A computer graphics interchange framework primarily for exchanging animation and visual effects data.
  • Textile Patterns:
    • WIF (Weaving Information File): Describes weaving patterns and structures.
    • Knitting Markup Language: Encodes knitting patterns in a structured format.
  • Scientific Data:
    • CDF (Common Data Format): Used for storing scientific data.
    • NetCDF (Network Common Data Form): Supports the creation, access, and sharing of array-oriented scientific data.
  • Photography and Imaging:
    • XMP (Extensible Metadata Platform): Used for embedding metadata in digital images and other media files.
  • Construction and Engineering:
    • LandXML: For civil engineering and land surveying data.
    • gbXML (Green Building XML): Facilitates the transfer of building data for analysis of energy and environmental performance.
  • Packaging and Retail:
    • BPL (Barcode Product Labeling): Encodes information for product packaging and labeling.
    • GS1 XML: Used for electronic business messaging, including product identification and tracking.
  • Typography and Font Design:
    • UFO (Unified Font Object): A format for storing font data.
    • SFNT (Spline Font): Encodes scalable font information.
  • Product Data Management:
    • PLMXML (Product Lifecycle Management XML): Used for sharing product data across PLM systems.

• Fri, Jul 12, 2024. Carlton's TDS session #markdown
  • Always create a new venv via VS Code when starting a training session. Helps reproduce issues (though I could use Colab instead)
  • Create an empty .ipynb notebook and double-click it. That's another way (though slower) to open a Jupyter notebook

• Wed, May 22, 2024. Predibase lets you run fine-tuned models at the same price, on a per-token basis. 25c/MTok up to 21B models. That's sames as Claude 3 Haiku, but with fine-tuning. #markdown #pricing

• Sun, Apr 28, 2024. ESLint's new flat configuration does not support package.json #markdown

• Sun, Apr 14, 2024. Outlines generates structured test with LLMs. It uses the ⭐ logit_bias trick to limit choices in output. See get_choice() #llm-ops #markdown

• Thu, Apr 11, 2024. VS Code #ai-coding-tools #code-agents #github #markdown
  • Select any code and command Explain this to understand the code
  • %something in command bar searches ACROSS files for a term. Exactly like Ctrl+Shift+F
  • Copilot has an Inline Chat: Start in Terminal (that needed me to unbind Ctrl+I in bash to work)
  • Ctrl+2 opens a second window on the side. Ctrl+1 goes back to the first window
  • Terminal: Open Detected Link lets you scroll through detected (file) links in terminal
  • Terminal sticky scroll is transparent. (But Terminal stick scroll isn't working for me.)
  • Copilot uses last 10 commit messages, Jupyter notebook kernel state (variables) as additional context
  • 1.88: supports locked scrolling to sync scrolling of side-by-side windows

• Fri, Mar 29, 2024. Rust is non-trivial. Inspired by We are under DDoS attack and we do nothing, I "wrote" a small binary that serves a parquet file as JSON. It failed and I couldn't fix it. #github #markdown

• Fri, Mar 29, 2024. spleeter is a better alternative to demucs. Splits audio into #markdown

• Wed, Mar 27, 2024. Representational engineering lets you control LLM output based on preference on the fly. #future #llm-ops #markdown #models #prompt-engineering

• Wed, Feb 28, 2024. SourceGraph is the default platform for private code completion & search #ai-coding-tools #code-agents #future #github #markdown

• Sun, Feb 25, 2024. demucs does an EXCELLENT job of splitting songs into drums, bass, vocals and others #ai-coding-tools #dev #markdown

• Sat, Feb 24, 2024. Architecture.md is an emerging standard #future #markdown

• Fri, Feb 23, 2024. htmz is a fantastic way to load HTML into elements! #future #html #markdown #web-dev

• Fri, Feb 23, 2024. Andrej Karpathy's summary of sharp edges in tokenization uses tiktokenizer to explain: #markdown
  • Why LLMs can't be used for spelling
  • Why LLMs are better at English than other languages
  • Why LLMs are bad at math
  • Why SolidGoldMagiKarp is a single token
  • Why trailing spaces are bad
  • Why YAML tokenizes more efficiently than JSON

• Mon, Feb 19, 2024. Python fastcore has decorators like @typedispatch, Self, etc. #markdown #python

• Fri, Feb 9, 2024. Fine-tuning session by Dan. Notebook #gpu #llm-ops #markdown
  • Example of fine-tuning Mistral. Consumed 28 computes ($2.8)
  • Axlotl is what the top fine-tuned LLMs are trained on
  • Deepspeed provides distributed training
  • Flash attention lets data stay on GPU
  • Sample packing packs samples of different lengths into equal length tensors

• Sun, Feb 4, 2024. Stephen Osserman's Observables have some nice notes. #markdown
  • Visualizing partial election results
  • D3 Force Dilemmas: Data Distortion

• Sun, Feb 4, 2024. Sandra Becker's 30 day D3 course #future #markdown

• Sat, Jan 13, 2024. Marimo is a reactive alternative to Jupyter notebooks that saves files as pure Python. #github #markdown #python

• Sun, Jan 7, 2024. mixtral-offloading cleverly loads only the model layer required at any point, letting you run Mixtral 8x7b on Colab Free and on 16GB GPUs. This notebook runs on Colab Free too. #cloud #gpu #markdown

• Sat, Dec 23, 2023. DPO is a simpler alternative to RLHF for fine-tuning. Several HuggingFace models use DPO for training #gpu #markdown

• Sat, Dec 16, 2023. Github Copilot's new features makes your entire workspace or a specific file its context. It also auto-writes your commit messages and PR descriptions. #github #markdown #prompt-engineering

• Sun, Dec 10, 2023. v0.dev creates web pages from code. Example. #dev #future #github #html #markdown #web-dev

• Thu, Dec 7, 2023. 12 Tuning Strategies for RAG #markdown #optimization #prompt-engineering

• Thu, Dec 7, 2023. CodeAnt.ai is a VS Code plugin to detect code smells, refactor for modularity, to write docstrings and unit tests #ai-coding-tools #code-agents #github #markdown

• Sun, Dec 3, 2023. Convert PDF to markdown with marker - an improvement over nougat. #document-conversion #markdown

• Sat, Nov 25, 2023. Learn ML optimization approach - LLMOps. Have an evaluation framework with metrics like weights and biases or tensorboard. Helps figure out where fine tuning helps and where RAG does. Soon, this will become important. #llm-ops #markdown #optimization

• Mon, Nov 20, 2023. jsonrepair can fix JSON fairly well. jsonformer wraps HuggingFace models to produce JSON. Ref #huggingface #markdown

• Sun, Nov 12, 2023. Pydantic is all you need drives towards code = data = text! #ai-coding-tools #future #markdown

• Sat, Nov 4, 2023. 04 Nov 2023: Sparsify quantizes models #markdown

• Mon, Oct 23, 2023. 23 Oct 2023: Bits & Bytes, QLORA, llama.cpp, BigDL, are ways of quantizing models that you can run directly in Google Colab. Try them. Also try OpenVino. #ai-coding-tools #github #gpu #markdown