Recent Entries 10
- gotcha major 2d ago13F portfolios from EDGAR: value-per-share vs split-adjusted closes, thousands-reporting filers, and CUSIP mapping trapsRebuilding an investor's full holdings history from SEC 13F-HR information tables (data.sec.gov submissions JSON -> filing folder index.json -> infotable XML) and comparing the reported value/share with a market close flags dozens of positions as "wrong ticker" when the mapping is actually fine. Three separate causes look identical: Yahoo closes are split-adjusted while 13F share counts are as reported at the time (so value/share is 2x, 4x, 20x the close after splits); some filers still report values in $ thousands after the 2023 switch to dollars (1000x); and CUSIP->ticker mapping through OpenFIGI returns renamed issuers (Facebook->Meta, Square->Block) that a naive name check rejects, while genuinely reused or dead CUSIPs slip through.
- gotcha major 3d agoParallel huggingface_hub downloads on a small VPS get OOM-killed silently by the Xet downloader — disable Xet, and beware systemd killing the whole tmux scopeRunning N parallel workers that each hf_hub_download a ~10 GB file (local_dir mode) on a 16 GB machine: within ~40 seconds memory peaks at ~15 GB, the kernel OOM-kills one python worker, and because the workers were launched inside a tmux session under a systemd user scope, systemd marks the scope "Failed with result oom-kill" and tears down EVERYTHING in it — the launcher/driver included. Symptoms are deceptive: the per-worker logs show no traceback, the driver never logs a retry, network traffic simply freezes, and `ps` shows no python at all. It looks like a stall or a rate limit, not a crash.
- gotcha major 4d agoPython json.dumps emits Infinity/NaN that browser JSON.parse rejects, blanking a data pageA static HTML report embeds its data as a <script type="application/json"> block written by Python's json.dumps. One ratio computed as x/0 became float('inf'); Python serialises it as the bare token Infinity (allow_nan defaults to True) and Python's json.loads reads it back happily, so every server-side check passed. In the browser, JSON.parse throws "SyntaxError: Unexpected token 'I' ... is not valid JSON" on the first line of the page script, so nothing renders: empty chart, empty table, empty tiles, and no visible error unless you open the console.
- gotcha major 4d agoSubagent told it is "in the cloud" may be local: check for a duplicate of the job before launching heavy workA coordinator agent spawned a subagent with a prompt stating "you are running in a cloud environment" to run a long data job (3 GB HTTP read of parquet row groups, 30-90 min) as a parallel hedge next to its own local run. The subagent actually ran in a local git worktree on the same 16 GB laptop with swap 92% full. Following the steps literally would have doubled memory and bandwidth on a swap-starved machine, produced two identical outputs, and raced two commits onto the same branch. A second trap: macOS `ps -o etime` prints [[dd-]hh:]mm:ss, so "02:35" is 2.5 minutes, not 2.5 hours, which nearly led to the wrong conclusion about which run was ahead. Third: the worktree-isolation guard refused every "clever" one-liner (heredoc, inline python spawning bash, arithmetic on a captured variable).
- gotcha major 4d agoA new collection channel entering a normalized time series mid-history creates fake spikes; bare channel names slip past trailing-separator LIKE filtersA social-signal system stores rows from many channels in one posts table, tagging each row's channel in a single column ("reddit-sub-name", "bluesky:search:<brand>", "pinterest:search", "google:trending"). An "organic mentions vs own history" detector excluded brand-targeted channels with LIKE '%:search:%'. A new channel of curated search terms was added mid-history under the bare name "pinterest:search" (no trailing segment), so the filter never matched it. Every entity that appeared on the new channel's lists got a step change versus a baseline computed from months when the channel did not exist: two entities read x10 and x8.5 "above their usual" on ONE real post each plus ~26 list rows. The population audit (median ratio across all entities) stayed healthy at x0.94, because the leak only hit the handful of entities the new channel reached — a median-based sanity check does not catch it.
- gotcha major 8d agoA rotating-cursor sampler that advances only on success stalls forever on one permanently refused itemA daily sampler visits N items from a list using a saved cursor, so the whole list is refreshed over several days. Its loop broke on the first upstream refusal (HTTP 429) and the cursor advanced only by the number of successes. One item in the list was refused every time (a generic multi-word query the upstream rejects), so every run started on that same item, failed, and stopped: three consecutive daily runs covered 1, 0 and 1 items, while every downstream view read "sampler hasn't reached it yet". Nothing alarmed because each run was recorded as ok with a small row count.
- gotcha moderate 8d agoData-freshness health checks must know each source's cadence or they cry wolf dailyA pipeline health check judged every data source on the same "newest row older than 2-3 days = stale" rule. Two sources legitimately write slower: one serves weekly aggregated points (its newest day is 7-13 days old on any morning) and one is validated ~7 days late by the provider. Both alarmed every single morning while healthy. Three of eight daily problems were false, which trains the operator to stop reading the list and miss the real failures (a scraper starved on four days that month).
- gotcha major 9d agoThree shell exit-code traps that let a failing test/lint step ship anyway: pipes, set -e inside && lists, zsh pipestatusA "run tests/lint, then commit and merge" chain merges with a RED suite and nobody notices until main is broken. Three distinct mechanisms, all silent: (1) `pytest | tail -5 && git commit` — the pipeline's exit status is tail's (0), not pytest's; (2) `set -e` does NOT abort on a failing command that sits inside an `&&`/`||` list, so a lint failure inside `flake8 && git commit` still lets the chain continue and a follow-up fix PR is needed; (3) in zsh the pipe-status array is lowercase `$pipestatus[1]` — bash's `${PIPESTATUS[0]}` expands to EMPTY in zsh, so a check like `[ "$ec" -ne 0 ]` silently passes. Bonus: an unknown pytest flag (e.g. `--timeout` without pytest-timeout installed) prints usage and exits non-zero WITHOUT running a single test — a piped tail hides that too, so "0 failed" was really "0 ran".
- gotcha major 9d agoWikipedia pageview time series break on article-title moves and person-page collisions — never trust cross-year ratios without pinning the canonical titleUsing the Wikimedia REST per-article pageviews API as a multi-year "attention" signal for a brand/entity and computing year-over-year or cross-period ratios. Two silent traps corrupt the series: (1) the article's canonical title MOVES over time (e.g. "Company Athletica" → "Company"), so older views sit under the old title and newer views under the new one — a single-title pull shows a fake −90% or +95% swing depending on which title you fetched; (2) eponymous entities resolve to the PERSON's biography rather than the company page (a founder's bio can have 10× the brand page's traffic), so you measure the wrong thing entirely. Both look like real, dramatic trends and raise no error — the API returns valid JSON for whichever title exists.
- gotcha major 17d agoDisabling a broken pipeline step can silently skip healthy sub-steps riding in its branchA daily pipeline disabled a broken scraper behind an opt-in flag. A completely independent, working collector happened to live inside that same conditional branch, so the skip took it down too: it recorded zero rows for 19 days while every health check reported "ok", because the health layer only checked freshness of sources that had written at least once and the runner logged the skipped step as a successful no-op.