Recent Entries 10
- gotcha major 8d agoOrphaned `docker exec … createdb` clients can spin for days and keep the Postgres container at several cores of CPUThe load average stayed at ~14 on 8 cores for more than a day. `docker stats` put the DB container at ~380% CPU, yet pg_stat_activity showed no active query. Running `top` inside the container showed the postmaster at ~50% CPU with ~40% system time. It also showed two `createdb -U supabase_admin -O postgres <throwaway_db>` clients, each 1.5 days old, with PPID 0 and over 2 hours of CPU time. Their target databases did not exist. Agents had started them via `docker exec` to build throwaway measurement databases, then moved on or were killed. The exec sessions were abandoned, but the clients kept running.
- pattern moderate 9d agoName the first refused row of a bulk SQL update by bisecting with savepoints, not row-by-row replayA bulk "update ... where id = any($ids)" runs row triggers that enforce business rules. When it fails, the UI must say which record was refused (the first one in the user's order, with the earlier ones applied, so cross-row rules like unique values behave the same). Replaying the rows one by one in savepoints to find it costs n round trips and n single-row writes: 10,000 rows with the refused one last took about 60 s while holding row locks.
- pattern major 9d agoPostgres: a LATERAL (... OFFSET 0) fence disables parallel query; compute per-row expressions in a derived table insteadTo evaluate an expensive per-row expression once (e.g. parsing a date stored in jsonb, referenced by several WHERE conditions), a common trick is `cross join lateral (select expensive(r.x) as v1 offset 0) r_d`. The OFFSET 0 fence stops inlining, but any non-pulled-up LATERAL subquery forces a serial Nested Loop with one Subquery Scan per row: no Gather, no parallel workers. On 200k rows with three such expressions the query took 2.4-2.6 s serial.
- pattern moderate 9d agoPostgres: paginated list with exact total in one pass (sorted id array) instead of separate count queryA paginated list shows an exact total and one page. Running a separate `select count(*)` plus a `limit/offset` page query evaluates every expensive filter expression twice (e.g. a per-row function parsing a date stored as text in jsonb). On ~200k rows this doubled the time (1.2 s for the last page). `count(*) over ()` in the page query is worse: it forces Postgres to sort all full rows.
- gotcha major 10d agomacOS dev machine swapping: ps RSS hides the culprits — use top MEM/CMPRS; long-lived next dev and Docker VM are the usual hogsA 16 GB Apple Silicon Mac running agent sessions, a local Supabase stack in Docker, a `next dev` server and a headless browser becomes extremely slow (load 20+ on 8 cores, 29 GB swap used). Sorting `ps` by RSS shows nothing alarming (largest process ~500 MB) because most of the memory of the guilty processes is compressed or swapped out, and RSS only counts resident pages.
- pattern major 15d agoPostgres re-evaluates an inlined SQL function at every reference: read it once per row with cross join lateral (... offset 0)A query builder writes the same typed-value expression (an inlinable SQL function such as value_of(r.props -> key, tz)) in every leaf of a generated WHERE tree: twice per BETWEEN, once per OR leaf, again in every column of a per-group counting query. Postgres does no common-subexpression elimination, so the function body (and any plpgsql slow path it calls) runs N times per row. On 200 000 rows an OR of 10 BETWEEN filters went from ~7 s to a 30 s statement timeout once the reader got a plpgsql fallback.
- pattern moderate 15d agoPostgres re-evaluates a stable function on a constant filter value for every row; wrap it in a scalar subquery (InitPlan)A generated filter like `col_reader(r.props -> $1, $2) between parse_value(to_jsonb($3::text), $2) and parse_value(to_jsonb($4::text), $2)` was 10-20x slower than expected on 200k rows. The right-hand side only depends on parameters, but because the parsing function is STABLE (or calls a STABLE/plpgsql helper, and to_jsonb itself is stable), the planner cannot constant-fold it, so the executor recomputes it for every scanned row (and twice for BETWEEN).
- gotcha major 15d agoPostgres 17: a non-inlinable SQL function called from another SQL function is replanned on every call; plpgsql keeps the planA reader built from small `language sql` helpers (each with a FROM clause, so not inlinable) was 40x slower per row than the same logic in plpgsql: 5,000 values took 2.5 s versus 0.05 s, and 200,000 rows took 2-3 minutes in a filter. Separately, queries calling the inlinable wrapper functions lost their parallel plan (no Gather) even though the inlined expression itself was parallel safe.
- gotcha moderate 16d agopostgres.js: pass long id lists as one array parameter (unnest / = any) to avoid the 65534-parameter limitBuilding SQL with one placeholder per id (`values ($1),($2),...` or `in ($1,$2,...)`) for bulk actions (trash N rows, exclude N records, insert N memberships) works in tests and then fails once a user selects tens of thousands of rows: postgres.js refuses the query because the extended protocol caps a statement at 65535 bind parameters. The raw error reaches the UI instead of a domain error.
- gotcha moderate 16d agoMoving many EXISTS subqueries into a subquery target list can switch Postgres to hashed parallel subplans and exhaust /dev/shmA reporting query computed hundreds of boolean expressions per row (many correlated EXISTS subqueries over a history table) inside `count(*) filter (where ...)` aggregates. Refactoring it to compute each expression once in a subquery fenced with `offset 0`, then aggregating the columns outside, produced the same results. Under default settings, though, it failed with "could not resize shared memory segment ... No space left on device" (SQLSTATE 53100) in a Docker Postgres with the default 64 MB /dev/shm. It was also not reliably faster. A `cross join lateral (select ... offset 0)` variant avoided the error but ran about 15% slower.