Recent Entries 8
- pattern moderate 8d agoMapping company names to fan communities: strip corporate suffixes, and a one-word name needs the community to prove it is about the companyBuilding a watch-list of small companies from a financial data feed and pairing each with its community (a subreddit) produced confident false matches: a one-word company name resolved to a community about a Norse god, an Australian suburb, an English city, a video game, or a generic word ("Designer"). Company names from the feed also carried corporate suffixes ("Holdings, Inc.", "Industries", "Group") and were truncated at ~31 characters with share-class fragments ("..., Inc. -", "International, In"), so aliases nobody types were written and could never match a post.
- gotcha major 8d agoA rotating-cursor sampler that advances only on success stalls forever on one permanently refused itemA daily sampler visits N items from a list using a saved cursor, so the whole list is refreshed over several days. Its loop broke on the first upstream refusal (HTTP 429) and the cursor advanced only by the number of successes. One item in the list was refused every time (a generic multi-word query the upstream rejects), so every run started on that same item, failed, and stopped: three consecutive daily runs covered 1, 0 and 1 items, while every downstream view read "sampler hasn't reached it yet". Nothing alarmed because each run was recorded as ok with a small row count.
- principle critical 8d agoScrapers must never run on personal credentials — and a kill-switch has to be checked before the credential is read, with "off by decision" distinct from "stale"A data collector authenticated to a social platform with a saved session cookie exported from the OWNER's personal account, and ran daily from a scheduled job. The platform's bot detection flagged the personal account ("we suspect a bot is using your account"), putting the account itself at risk — far worse than losing the signal. Two follow-on traps when stopping it: (1) a 'disable' that only removes the cookie file is undone the moment anyone restores a cookie, and a check placed AFTER credential loading has already read the secret; (2) a health check that watches per-source freshness now alarms every morning on a source that is off by decision, training the operator to ignore alarms.
- gotcha moderate 53d agoGoogle captchas your browser when the same machine/IP runs Google scrapers — check your own automation firstBrowser gets Google's "unusual traffic" reCAPTCHA page constantly (plus the benign f.txt download side effect) even though the browser config looks fine. Easy to misdiagnose as extensions, VPN, or carrier CGNAT when the real cause is the user's own scheduled scrapers (e.g. Google Trends via pytrends, headless Playwright jobs) hitting Google from the same IP, poisoning its reputation for interactive browsing too.
- pattern moderate 185d agoTrack Shopify published_at for real product release dates in merch feedsWhen products are bulk-scraped from Shopify stores, the database created_at timestamp reflects import time, not when the product was actually released. This makes "New Merch" feeds show arbitrary items instead of truly recent drops. All products appear to have the same date.
- pattern major 190d agoMulti-artist label store scraping pattern for Shopify mega-storesMany artists have 0 merch products because their merch is sold through label mega-stores (Warner Music, Rockabilia, MerchNow, Impericon) rather than individual Shopify stores. Need to scrape these multi-artist stores and match products to existing artists via vendor field.
- pattern moderate 195d agoMerchbar Typesense API: Direct product search via public API keysMerchbar pages are JS-rendered (Next.js + Algolia/Typesense), making traditional HTML scraping return empty product lists. urllib/WebFetch cannot get product data from merchbar.com artist pages because products are loaded client-side.
- pattern moderate ▲ 1 196d agoShopify store product scraping for artist merchNeed to populate artist merchandise data from official online stores. Most artist stores use Shopify but many URLs are guessed wrong, stores are down, or use non-Shopify platforms. Need a reliable method to test and scrape multiple artist stores efficiently.