Recent Entries 8
- gotcha major 4d agoA new collection channel entering a normalized time series mid-history creates fake spikes; bare channel names slip past trailing-separator LIKE filtersA social-signal system stores rows from many channels in one posts table, tagging each row's channel in a single column ("reddit-sub-name", "bluesky:search:<brand>", "pinterest:search", "google:trending"). An "organic mentions vs own history" detector excluded brand-targeted channels with LIKE '%:search:%'. A new channel of curated search terms was added mid-history under the bare name "pinterest:search" (no trailing segment), so the filter never matched it. Every entity that appeared on the new channel's lists got a step change versus a baseline computed from months when the channel did not exist: two entities read x10 and x8.5 "above their usual" on ONE real post each plus ~26 list rows. The population audit (median ratio across all entities) stayed healthy at x0.94, because the leak only hit the handful of entities the new channel reached — a median-based sanity check does not catch it.
- pattern moderate 6d agoPeriod roll-up pages from persisted daily rankings: count "days seen of days recorded", exclude list-everything boards, and give reused sections a no-write modeA daily anomaly report persisted each board's rows to a history table, and the operator then asked for weekly and monthly reports. Reusing the daily section builders for the period pages had two traps: those builders wrote their rows to the history table as "today's" record, so a period page rebuilt with a different window silently overwrote the daily record; and one board that lists every tracked entity every day (a universe list, not a ranking) made every entity a "seen every day" regular in the roll-up. Early in the history, "seen 3 days" also read as strong when only 3 days had ever been recorded.
- pattern moderate 6d agoMulti-horizon anomaly boards: one rule on 1/7/30-day buckets, with a stricter count floor for the day span and a report-only rerun flagA daily report had "week" and "month" views of an anomaly rule (an entity fires when a stream is above its own median) but no "day" view, and adding one naively either flooded the board with one-day counting noise or duplicated the rendering code with hard-coded period words ("this week" printed on the month board). Rebuilding the page for a same-day rerun also re-ran every network collector, doubling request loads on rate-limited or fragile accounts.
- pattern major 7d agoAttention screens find loud; the shape that paid was quiet: add a bottom-up board judged against each entity's own past and label dead communitiesA multi-source attention system had a dozen boards that all ranked entities by how much louder than usual they were. Reviewing the three cases that actually worked for the operator showed the opposite shape: small entities with no broad attention, whose own community was slowly tightening (more posts from the same people, more scarcity language). None of them ever reached the top of a board, so the system could not put the right kind of candidate in front of the operator. A second failure: the universe builder added communities that merely existed (newest post six years old, one private since spring) and every board read their silence as a quiet week.
- gotcha moderate 8d agoData-freshness health checks must know each source's cadence or they cry wolf dailyA pipeline health check judged every data source on the same "newest row older than 2-3 days = stale" rule. Two sources legitimately write slower: one serves weekly aggregated points (its newest day is 7-13 days old on any morning) and one is validated ~7 days late by the provider. Both alarmed every single morning while healthy. Three of eight daily problems were false, which trains the operator to stop reading the list and miss the real failures (a scraper starved on four days that month).
- principle major 8d agoScore a signal detector against its own draw universe and persist the boards humans readA signal system looked "positive" (+1.4% average 21-day forward return, 52% up) until compared with every ticker it draws from over the same weeks (+5.6%, 55% up): it underperformed its own source list. Separately, the human-facing summary board (the one the operator actually reads and would trade from) was rendered into HTML each day but never written to any table, so it could never be backtested even months later. And 96% of stored signals had no ticker attached, so they were counted as detections but could never be scored.
- pattern major 8d agoPortable `timeout` for macOS via process-group kill — and why the watchdog timer needs its own group toomacOS ships without `timeout`/`gtimeout`. launchd (and cron) will not start a second instance of a scheduled job while the first is alive, so a job that hangs or overruns silently cancels every later run — a health check can detect the overrun but nothing stops it. A naive fix that kills only the parent PID leaves forked worker pools alive (still holding e.g. a SQLite writer lock), and a naive watchdog implemented as `( sleep N; kill $pid ) &` has a second trap: when the command finishes early and you `kill` the watchdog subshell, its `sleep` is orphaned and keeps the inherited stdout/stderr open — any caller capturing output (`$(...)`, subprocess pipes, tests) then blocks for the full grace period.
- pattern major 185d agoBulk importing StackExchange XML data dumps into SQLite with streaming XML parsingNeed to import tens of thousands of Q&A entries from StackExchange data dumps (archive.org/details/stackexchange) into a SQLite database. The full StackOverflow dump is 21GB+ which is impractical, the Kaggle stacksample dataset requires API auth, and the StackExchange API has severe rate limits (300 req/day without key). Need an efficient approach to get 50K+ high-quality entries.