#database | Anand S - Things I Learned

Sun, Feb 23, 2025. Rill Data is an interesting BI tool based on DuckDB. It auto-generates a dashboard given a dataset. #database

Sun, Feb 23, 2025. It's possible to assign "variables" in SQL (notably in DuckDB). Here's an example: #database

WITH
  sessions AS (FROM events SELECT COUNT(DISTINCT session_id) AS value),
  pages AS (FROM events SELECT COUNT(*) AS value)
FROM sessions, pages
SELECT sessions.value / pages.value AS pages_per_session;

Sun, Feb 2, 2025. ibis is a Python library that works with multiple dataframe backends like DuckDB, Polars, and Pandas. #database

Sun, Jan 5, 2025. Every few years, something comes along trying to replace relational databases and SQL, and gets absorbed. YouTube #database
- Key value stores. People soon realize they need more features, e.g. indices.
- MapReduce systems. Most MapReduce vendors put SQL on top of SQL. Then the Hadoop market crashed. (But HDFS, S3, distributed storage systems are a good idea)
- Document Databases. JSON. SQL absorbed that. SQLite 3.45+ supports even JSONB. DuckDB, of course, has JSON.
- Column Databases. Again, these introduced SQL.
- Graph Databases. SQL:2023 introduced graph queries via SQL/PGQ (Property Graph Queries). DuckPGQ beats Neo4J
- Array Databases. SQL:2023 adds SQL/MDA which allows for matrix operations. But specialized databases might make sense in this category.
- Vector Databases. Every DB is adding support for this.

Tue, Nov 26, 2024. LLMs may be good at derived data generation. For example, given a database schema, what derived columns would be useful? What derived views would be useful? #database #llm-ops

Fri, Nov 22, 2024. DuckDB has an LLMs.txt. #database #github #llm-ops #markdown Today, 38 repos on GitHub support it

Fri, Oct 4, 2024. There's a movement towards embeddable databases on the cloud. #cloud #database
- MotherDuck is hosted DuckDB.
- Turso is hosted SQLite (with local sync, multi-tenant)
- StarBase DB is SQLite with an API on top of Cloudflare Durable Objects.

Sat, Mar 30, 2024. sqlite-schema-diagram generates schemas for SQLite databases using Graphviz #database

Sat, Feb 17, 2024. Discussion with Vinu Yamunan #dev #database
- Databuck by FirstEigen. Autolysis plus monitoring
- Quality council has the data steward (maintainer of each dataset) coming together with the uses on a weekly basis to understand what quality problems to users are facing. Data owners jaundice at a lower frequency to get an understanding
- #TODO Automate rules for data quality in our projects and intranet
- Convert a config rule into business language. Explain SQL. These are good use cases for llm's
- Graph DBs are powerful for flexible data structures, but query generation needs AI or expertise. Check the Neo4J language cypher
- Explore storing SAME data in relational DBs AND in graph DBs / document DBs for different use cases
- Dallas rocketry challenge. Build a rocket that can take an egg to 800 feet exactly and land without breaking it

Tue, Jan 16, 2024. Lilac is an LLM-based data curation tool. Use it to search by concept (e.g. PII, duplicates, etc.) and then drop/update the results. #ai-coding-tools #database #llm-ops

Thu, Dec 21, 2023. Google Knowledge Graph ID powers the Knowledge Graph. If it begins with /m/ it's the same as the FreeBase ID. This is now available as WikiData. e.g https://www.wikidata.org/wiki/Property:P2671 #database

Tue, Nov 7, 2023. Julius.ai queries structured data. #database #future