#document-conversion

function HorizontalRule()
  return pandoc.RawBlock('openxml', '<w:p><w:r><w:br w:type="page"/></w:r></w:p>')
end

const blobToDataURL = (blob) =>
  new Promise((res, rej) => {
    const r = new FileReader();
    r.onload = () => res(r.result);
    r.onerror = () => rej(r.error);
    r.readAsDataURL(f);
  });

• Policy-as-code app. Checklist from doc. Apply checklist to data / doc inputs.
• Code similarity checker library based on TDS Project evals.

• "Write a Python script to extract a page from a PDF file and save it." Followed by "Write minimal code. Drop error handling."
• "Write a Python script to pass a PDF file to an LLM for OCR and print the result. Use this code sample... [PASTED CODE]." Followed by "Write minimal code. Drop error handling."

• How to parse PDF docs for RAG is a useful OpenAI cookbook with a GPT 4o prompt

• --no-highlight skips code-highlighting. --highlight=pygments adds Pygments styling
• --wrap=none doesn't wrap the content in a single block
• --number-sections adds section numbering (<h2>1. Introduction</h2>)
• --shift-heading-level-by=NUM – shift all headings by NUM levels (e.g., start at <h2> instead of <h1>)
• pandoc -f markdown-auto_identifiers drops the auto-identifiers extension that generates id=... for each heading
• pandoc -f gfm uses GitHub flavored Markdown. Run pandoc --list-extensions=gfm to identify the extensions it uses.
• Pandoc's Markdown extension examples are quite extensive.
• Auto-enabled GFM extensions:
  • alerts: GitHub-style callouts (info, tip, warning) via > [!TYPE] blocks.
  • autolink_bare_uris: Turns bare URLs into links, without needing <...>.
  • emoji: Parses :smile:-style codes into Unicode emoji characters.
  • footnotes: Enables footnote syntax with [^id] and definitions at the bottom.
  • gfm_auto_identifiers: Uses GitHub’s heading-ID algorithm: spaces → dashes, lowercase, removes punctuation.
  • pipe_tables: Enables table.
  • raw_html: Raw HTML is unchanged.
  • strikeout: Enables strikethrough with ~~text~~.
  • task_lists: Parses - [ ] and - [x] items as checkboxes.
  • yaml_metadata_block: YAML front matter for document metadata, e.g. <title>
• GFM extensions worth enabling:
  • ascii_identifiers: Strips accents/non-Latin letters in automatically generated IDs.
  • bracketed_spans: [Warning]{.alert} becomes <span class="alert">
  • definition_lists: Term\n: Definition text becomes a definition list
  • fenced_divs: ::: {.note} block creates a <div class="note">...</div>
  • implicit_figures: Standalone images become <figure> with <figcaption>.
  • implicit_header_references: [Section] is treated as [Section][#section]
  • raw_attribute: <b>bold</b>{=html} is inserted as HTML
  • smart: Converts straight quotes to curly, -- to en-dash, --- to em-dash, ... to ellipsis.
  • subscript & superscript: E.g. H~2~O and E = mc^2^

• Useful for: Document conversion, Thumbnail generation, Text extraction, Merging / splitting documents

• Copy an entire issue (with history) from GitHub, Gitlab, or JIRA
• Copy an entire PR (with code changes) from GitHub, Gitlab, or Bitbucket
• Copy CI/CD logs from GitHub Actions, Gitlab CI, Azure DevOps, etc.
• Copy entire conversation thread in Gmail or Discourse, Service now etc.
• Copy product reviews from Amazon, Shopify, etc.
• Copy page(s) from wikis and content sites like Wikipedia, StackOverflow, etc.
• Copy survey responses from Google Forms, Typeform, etc.
• Copy all interactions with a contact (including interactions, proposal history) from HubSpot or Salesforce
• Copy transcripts from Zoom, Teams, Google Meet, etc.
• Copy as Markdown from Word, GDocs, PDF or HTML
• Copy the summary of an analysis as well as all key metrics from any dashboard
• Copy SAP invoices
• Copy JDs, CVs, and reviews from Workday, BambooHR, DarwinBox, etc.
• Copy design specs, component libraries, and style guides from Figma, Miro, etc.
• Generated with the help of ChatGPT -- link not working

• Marker from DataLab seems comparable but is CC-BY-NC-SA.
• MinerU convert medical textbooks to Markdown well.
• Gemini Flash may be more cost effective and better

• uvx files-to-prompt
• npx git-ingest
• ingest - written in Go, only Mac/Linux binaries

• PyMuPDF4LLM uses MuPDF. Requires PyTorch.
  • PYTHONUTF8=1 uv run --with pymupdf4llm python -c 'import pymupdf4llm; h = open("pymupdf4llm.md", "w"); h.write(pymupdf4llm.to_markdown("$FILE.pdf"))'
• markitdown from Microsoft. PDF via PDFMiner, DOCX via Mammoth, XLSX via Pandas, PPTX via Python-PPTD, ZIP, etc.
  • PYTHONUTF8=1 uvx markitdown $FILE.pdf > markitdown.md
• Docling by IBM. Unable to install via pip on Windows AND on Linux.
• MegaParse uses libreoffice, pandoc, tesseract-ocr, etc. Requires OpenAI API key.

• Jekyll - Ruby - 2008
• MkDocs - Python - 2014
• GitBook - JavaScript (Node.js) - 2014
• MkDocs Material - Python (MkDocs-based) - 2016
• Docsify - JavaScript - 2016
• MdBook - Rust - 2017
• Antora - JavaScript (Node.js) - 2017
• Docusaurus - JavaScript (React) - 2017
• JupyterBook - Python - 2019
• Keenwrite - Java - ~2019
• Honkit - JavaScript (GitBook fork) - 2019
• Nextra - JavaScript (Next.js) - 2020
• Astro - JavaScript/TypeScript - 2021
• Hugo Book - Go (Hugo-based) - ~2020
• Clowncar - JavaScript/Node.js - ~2021
• Quarto - R and Python - 2022
• Starlight - JavaScript/TypeScript - 2023

• 04 Oct 2024. PDF-Extract-Kit does PDF layout, formula, table, and OCR extraction using various models.
• 04 Oct 2024. llmsherpa extracts PDF layout, tables, not OCR