https://appcoding.com/tags/databases/ Recent content in Databases on App Coding Hugo en-us Fri, 09 Oct 2026 00:00:00 +0000 https://appcoding.com/counterfactual-store-a-database-that-stores-alternatives-not-just-facts/ Fri, 09 Oct 2026 00:00:00 +0000 https://appcoding.com/counterfactual-store-a-database-that-stores-alternatives-not-just-facts/ <p>One direction that feels genuinely different from the <a href="https://appcoding.com/20-primitive-project-ideas-tiny-go-engines-that-do-the-essential-10-of-redis-kafka-and-etcd/">earlier list of 20 primitives</a> is a Counterfactual Store. Adjacent ideas certainly exist. Provenance databases, event sourcing, probabilistic databases, simulation systems and version control all touch pieces of it. But as a deliberately tiny infrastructure primitive, there&rsquo;s no established category built around this exact abstraction.</p> <p>Normal databases answer &ldquo;what is true?&rdquo; Temporal databases, and <a href="https://appcoding.com/temporal-kv-a-key-value-store-that-remembers-history/">Temporal KV</a>, add &ldquo;what was true?&rdquo; A Counterfactual Store adds a third basic question: what would be true if X were different?</p> https://appcoding.com/temporal-kv-a-key-value-store-that-remembers-history/ Fri, 09 Oct 2026 00:00:00 +0000 https://appcoding.com/temporal-kv-a-key-value-store-that-remembers-history/ <p>Temporal KV is a key-value database where a key doesn&rsquo;t just have a current value. The database automatically remembers how that value changed over time.</p> <p>In an ordinary KV store, you might write <code>price = 100</code>, then later <code>price = 110</code>, then <code>price = 125</code>. Ask for <code>price</code> and you get 125. The earlier values have normally been overwritten. In a Temporal KV store, all three states stay logically accessible. You can ask for the current value and get 125, or ask &ldquo;what was price at 10:00 yesterday?&rdquo; and get 100. The basic abstraction stops being key → value. It becomes key + time → value.</p> https://appcoding.com/a-database-where-every-value-remembers-its-source-confidence-and-extraction-time/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/a-database-where-every-value-remembers-its-source-confidence-and-extraction-time/ <p>Somebody pastes a population figure into a shared spreadsheet: 1,234,567. It might have come from a census table, a Wikipedia infobox, a city press release or a model&rsquo;s summary of one of those, and three weeks later nobody can say which. The paste kept the digits. Everything that made them believable stayed in a browser tab that closed long ago.</p> <p>A value without provenance is a rumor with a data type.</p> https://appcoding.com/a-feature-store-small-enough-to-embed-point-in-time-reads-for-local-and-edge-ml/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/a-feature-store-small-enough-to-embed-point-in-time-reads-for-local-and-edge-ml/ <p>A churn model looks excellent in the notebook and mediocre in production. The model isn&rsquo;t broken. The training table computed <code>orders_7d</code> with a rolling window over the full history, so some rows counted orders placed after the label date. The serving code, written later in another language, counts a calendar week instead of seven days. One feature, two definitions, and a quiet look into the future. Those are the two bugs a feature store exists to prevent: training-serving skew and label leakage.</p> https://appcoding.com/a-schema-diff-that-warns-which-migration-will-lock-a-40-million-row-table-or-lose-data/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/a-schema-diff-that-warns-which-migration-will-lock-a-40-million-row-table-or-lose-data/ <p>The diff in the pull request is one line: <code>ALTER TABLE orders ALTER COLUMN total TYPE bigint;</code>. It reads like a rounding fix, so the reviewer approves it, and the SQL is correct. On a table with 41 million rows it&rsquo;s also a full rewrite under a lock that stops every read and write until it finishes. A diff shows what changes, never what the change costs.</p> <p>A schema diff worth using would print consequences instead of SQL, in sentences like &ldquo;this rewrites a 41-million-row table and blocks every read and write&rdquo; or &ldquo;the rollback is impossible without a backup&rdquo;. Those sentences depend on three things: the statement, the engine and its version, and the size of the table it lands on. With all three you can post a risk report on every migration pull request. With only the first you have a linter.</p> https://appcoding.com/a-telemetry-database-that-forgets-on-purpose-keeps-rollups-and-anomalies-instead-of-raw-rows/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/a-telemetry-database-that-forgets-on-purpose-keeps-rollups-and-anomalies-instead-of-raw-rows/ <p>At 300 requests a second, one latency metric adds about 26 million rows a day. Nobody will read most of them again. What people ask for a month later is the p99 per route for last Tuesday, and the one request that took nine seconds. A rule that deletes rows after 30 days throws away both answers. Keeping every row forever pays storage bills for data nobody reads.</p> <p>The telemetry store worth building decides at write time what has to survive. It folds everything else into coarser buckets on a schedule and keeps the few raw events that carry information, whole. Forgetting becomes part of the schema, declared in a policy, instead of a cron job that deletes by age. This is the argument for <a href="https://apicoding.com/the-best-small-infrastructure-tools-reduce-data-near-the-source-instead-of-storing-more/">reducing data near the source</a>, applied to time series.</p> https://appcoding.com/a-tiny-sql-engine-for-json-streams-fits-between-jq-and-a-stream-processing-cluster/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/a-tiny-sql-engine-for-json-streams-fits-between-jq-and-a-stream-processing-cluster/ <p>Your service writes one JSON object per line to stdout, and you want the average <code>cpu</code> for host <code>web-3</code> over the last five minutes. jq can&rsquo;t answer that, because it forgets each object once it has printed it and has no idea what &ldquo;five minutes&rdquo; means. You can fake both with <code>jq -n 'reduce inputs as $e (...)'</code> plus a timestamp comparison, and you&rsquo;ll have a script nobody wants to touch twice. DuckDB answers the question in one line, but only against a file as it exists right now. ksqlDB, Materialize, RisingWave, Arroyo and Flink SQL answer it continuously, once you&rsquo;ve deployed a service and agreed to operate it.</p> https://appcoding.com/an-embedded-sketch-database-billions-of-events-megabytes-of-storage-error-bars-on-every-answer/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/an-embedded-sketch-database-billions-of-events-megabytes-of-storage-error-bars-on-every-answer/ <p>Counting unique visitors exactly means remembering every visitor. Say 6 million distinct IPv4 addresses hit a checkout service today. Packed at 4 bytes each, that set is 24 MB before any hash table overhead, and you need one per service, per region, per day you want to compare. Redis answers the same question with a HyperLogLog in about 12 KB per key, at a standard error of about 0.81%.</p> <p>The thin part is the database around it: a SQLite-shaped file of sketches with time buckets, rollups and a SQL front door (check before you start; this space moves). The version worth building stores nothing but mergeable sketches, grouped by dimension and time bucket, and every aggregate returns its error bound next to the value. You never keep the events. It&rsquo;s the extreme case of <a href="https://apicoding.com/the-best-small-infrastructure-tools-reduce-data-near-the-source-instead-of-storing-more/">reducing data near the source</a>: keep the answer, drop the rows.</p> https://appcoding.com/context-window-management-as-a-database-ask-for-12000-tokens-get-the-most-useful-ones/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/context-window-management-as-a-database-ask-for-12000-tokens-get-the-most-useful-ones/ <p>An agent is ninety tool calls into a refactor. Step 12 was a grep that returned 52 KB, and all of it is still in the prompt. The user&rsquo;s instruction from step 1, &ldquo;never touch db/migrations&rdquo;, fell out of the window ten steps ago, because the framework drops the oldest messages first. The agent edits a migration. Nothing malfunctioned; the truncation rule did exactly what it was written to do.</p> https://appcoding.com/database-to-database-sync-for-local-first-apps-lives-or-dies-on-conflict-rules/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/database-to-database-sync-for-local-first-apps-lives-or-dies-on-conflict-rules/ <p>Take a field technician who edits a work order on a tablet in a basement with no signal. Upstairs, the office edits the same order from a desk. Both hit save at 10:42 by their own clocks, and one of those clocks is ninety seconds off. When the tablet finds a signal, the sync layer has to produce one order that both people can live with. That decision is the product. Offline-first apps are <a href="https://appcoding.com/building-offline-first-mobile-apps-is-harder-than-it-looks-and-worth-it/">harder than they look</a>, and a large share of the difficulty sits in this one step.</p> https://appcoding.com/duckdb-already-queries-every-file-in-a-folder-so-build-the-part-that-infers-the-joins/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/duckdb-already-queries-every-file-in-a-folder-so-build-the-part-that-infers-the-joins/ <p>Someone hands you a zip: <code>customers.csv</code> from the CRM, a JSON dump from the billing API, and a SQLite file from an internal app nobody has touched in two years. DuckDB will make all three queryable in a few lines of SQL, with no import step (<code>SELECT * FROM 'customers.csv'</code> works as written). Then you spend the afternoon working out which column joins to which. The CRM says <code>customer_id</code> and stores <code>00123</code>. The billing API says <code>customerId</code> and stores <code>123</code>. The SQLite table has a column called <code>customer</code> holding the text <code>123</code>. Same customer, three spellings.</p> https://appcoding.com/embedded-vector-search-is-crowded-what-a-one-file-vector-database-needs-to-stand-out/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/embedded-vector-search-is-crowded-what-a-one-file-vector-database-needs-to-stand-out/ <p>You have 40,000 support articles and an app that should offer &ldquo;similar articles&rdquo;. Embedded at 768 dimensions in float32, that&rsquo;s 40,000 × 768 × 4 bytes, about 123 MB of vectors. The standard advice is to stand up a vector database: a service, a port, credentials, a client library and a backup story for 123 MB. A file would do.</p> <p>Below a few million vectors the server is the overhead, which is why the embedded field filled up fast. So the useful question is what a newcomer brings that the free libraries don&rsquo;t. My answer is one sharp edge you can state in a sentence, plus measurements the others leave you to make yourself.</p> https://appcoding.com/facts-should-expire-a-database-that-tells-an-ai-agent-when-its-knowledge-has-gone-stale/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/facts-should-expire-a-database-that-tells-an-ai-agent-when-its-knowledge-has-gone-stale/ <p>An agent keeps a memory table, and two rows matter for today&rsquo;s task: a supplier&rsquo;s headquarters city, written eleven months ago, and a freight quote, written three hours ago. A <code>SELECT</code> returns both in the same shape. The headquarters is probably still right. The quote may already be wrong. Nothing in either row says so, and the agent repeats both in the same flat tone.</p> <p>A row has no idea how old its truth is. It might carry a <code>created_at</code>, and a careful person can compute an age from it, but the decay rate belongs to the kind of fact, not to the table. A three-week-old headquarters address is fine. A three-week-old shipping rate is fiction. Server health goes bad in half a minute. No timestamp column holds that knowledge, and nothing forces a query to read it anyway.</p> https://appcoding.com/give-every-database-query-a-budget-and-fail-the-build-when-it-scans-too-much/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/give-every-database-query-a-budget-and-fail-the-build-when-it-scans-too-much/ <p>A developer changes a login lookup to <code>WHERE lower(email) = lower($1)</code> so that sign-in stops being case-sensitive. Every test passes. The btree index on <code>email</code> can&rsquo;t serve a query that wraps the column in a function, so Postgres reads the whole users table, and on a test database with forty rows that&rsquo;s the cheapest plan anyway. The change ships. A week later someone notices login latency tracking the signup count. The fix is a one-line expression index that would have taken a minute if anything had objected during review.</p> https://appcoding.com/incremental-view-maintenance-small-enough-to-embed-turns-dashboard-queries-into-lookups/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/incremental-view-maintenance-small-enough-to-embed-turns-dashboard-queries-into-lookups/ <p>A dashboard tile shows today&rsquo;s US traffic, so every page load runs <code>SELECT count(*) FROM events WHERE country = 'US'</code> over a table that grew by a few hundred thousand rows since yesterday. SQLite answers correctly. It also walks every matching index entry to do it, because its b-trees don&rsquo;t keep counts, and it does the same walk for the next viewer and the one after. Between two page loads the answer moves by a handful of rows, yet each load pays for all of them.</p> https://appcoding.com/redis-semantics-without-redis-an-embedded-state-engine-for-counters-leases-and-ttls/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/redis-semantics-without-redis-an-embedded-state-engine-for-counters-leases-and-ttls/ <p>A command-line tool should run only one copy of its nightly export at a time. A small web app wants to cap each user at 60 requests a minute. A desktop app needs to remember when it last synced. All three are state: an integer, a timestamp, an owner. All three usually get solved by adding Redis, because Redis has the verbs for them: <code>INCR</code>, <code>SET NX PX</code>, <code>EXPIRE</code>. The app needed the verbs and got a server along with them, a port and a persistence config included.</p> https://appcoding.com/spreadsheet-recalculation-inside-a-database-derived-values-that-track-their-own-dependencies/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/spreadsheet-recalculation-inside-a-database-derived-values-that-track-their-own-dependencies/ <p>A line item&rsquo;s quantity goes from 4 to 5. The order subtotal should change, and so should the tax, the order total, the customer&rsquo;s lifetime value and the regional revenue figure on a dashboard. Which of those actually update depends on which code path made the edit, because each one gets recomputed somewhere different: the checkout handler, the invoice renderer, a nightly job, a SQL view somebody wrote for finance. Every copy of the formula is another chance for two screens to disagree.</p> https://appcoding.com/sql-and-key-value-over-the-same-bytes-an-old-database-idea-worth-making-small/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/sql-and-key-value-over-the-same-bytes-an-old-database-idea-worth-making-small/ <p>A sensor node writes a reading every few seconds. The firmware wants one thing from storage: put a record by key, get it back by key, as fast as the flash allows. The dashboard on the gateway wants something else, like average temperature by sensor by hour, joined to a table of sites. The usual answer is two databases with a translation job between them, a key-value store on the device and SQLite or Postgres upstream. Then comes a year of debugging the translation: a unit that changed, a row that arrived twice, a field the device encodes one way and the gateway reads another.</p> https://appcoding.com/store-changes-instead-of-current-rows-an-embedded-database-built-around-history/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/store-changes-instead-of-current-rows-an-embedded-database-built-around-history/ <p>A support ticket asks two plain questions: which plan was customer 4411 on when the August invoice went out, and who moved them off it? The <code>customers</code> table can&rsquo;t answer either. It holds today&rsquo;s row. The audit table someone added two years ago holds most of the story, apart from the week a data fix ran a bulk <code>UPDATE</code> in raw SQL and skipped the ORM callbacks that write the audit rows.</p> https://appcoding.com/verified-absence-is-evidence-a-database-for-recording-that-you-checked-and-nothing-was-there/ Mon, 05 Oct 2026 00:00:00 +0000 https://appcoding.com/verified-absence-is-evidence-a-database-for-recording-that-you-checked-and-nothing-was-there/ <p>An auditor asks you to show that customer 1187 was screened against a sanctions list on March 3. The screening job writes a row when it finds a match, and customer 1187 has no rows. That fits two histories: the job ran and found nothing, or the job never ran. The database can&rsquo;t tell them apart, so you hold no evidence either way.</p> <p>This is the closed-world assumption doing what SQL has always done: a row that doesn&rsquo;t exist is false. RDF and OWL went the other way with an open-world assumption, where a missing statement means unknown. Neither has a place for the sentence you actually want to store: I looked here, in this way, at this time, and found nothing.</p> https://appcoding.com/altsql-gives-iot-devices-a-15-kb-database-that-keeps-working-when-the-link-drops/ Fri, 02 Oct 2026 00:00:00 +0000 https://appcoding.com/altsql-gives-iot-devices-a-15-kb-database-that-keeps-working-when-the-link-drops/ <p>Anyone who has built the software side of a connected product knows the integration job. The device keeps its readings and settings in one format and the gateway or cloud service wants those same values as rows in a SQL database. Somebody writes a converter. Then somebody has to keep that converter alive for as long as the device stays in the field, which for industrial equipment or embedded systems can easily be ten years. The converter has to handle offline periods, retries, out-of-order updates, and the device&rsquo;s own data format changing over time.</p> https://appcoding.com/precomputing-keeps-dashboard-answers-ready-in-a-sqlite-file-as-the-data-arrives/ Fri, 02 Oct 2026 00:00:00 +0000 https://appcoding.com/precomputing-keeps-dashboard-answers-ready-in-a-sqlite-file-as-the-data-arrives/ <p>Most dashboards ask the same few questions all day long: requests per endpoint, month-to-date usage for one customer, error rates by status code. The usual setup recounts raw rows from scratch on every refresh, or ships every row to a hosted analytics service that bills by the gigabyte. A dashboard that needs to answer quickly on demand either waits for a database query to finish, or pays for a service that stores everything.</p>