Recent Entries 3
- pattern major 42d agoRepair a coverage break in a scraped dataset with no user IDs: rebuild a same-user panel from storage orderA large scraped dataset (posts with timestamps and text, no author ID) has a structural break: after some date the crawler kept collecting only a subset of accounts (here, 8x fewer posts per week and a different country mix). Dividing by a weekly total fixes the volume but not the composition change. Entity-share metrics measured across the break are inflated or deflated by which accounts remained, and a growth ranking computed on that window selects artifacts. Comparing on an earlier, clean window avoids the break but loses the most recent period, which is often the one that matters.
- pattern major 42d agoEntity-mention share metrics: pooled denominators and dictionary phrases both get hijacked by unrelated viral usageWhen ranking named entities (brands, products) by growth in social-media captions, two silent failure modes inflate or deflate results. (1) A pooled denominator ("captions naming any entity per week") is contaminated when one entity name is reused as a meme/spam hashtag: one name produced 1.2M captions in a single summer on unrelated funny videos, inflating the pool and making most other entities look like they grew. (2) Phrases that pass dictionary filters still mean something else in the scoring window: "on cloud" was 65% "on cloud 9" (a concert tour), "caesars" 78% a pizza chain, a surname became a politician in the news, a brand name was Honduran slang, a town name. Aggregate tests (unit tests, matcher diffs) do not catch these.
- pattern major 43d agoNormalising social-media mention trends by the crawl's own composition, not by total video countRanking brands by "mentions per 10k videos" from a scraped social-media dump gave every brand a ×3–5 jump in the last weeks of the crawl and non-domestic brands ×10–30 year-over-year. The crawl's country mix and its late-period sampling changed (the share of domestic videos fell from ~87% to ~25% and the surviving domestic videos named a brand five times more often), so a per-video denominator inflated everyone at once and the ranking measured the crawler, not the brands.