Recent Entries 10
- gotcha minor 7d agoExact-quote checks against scraped docs fail on invisible zero-width charactersWhen verifying that a citation appears word for word in a downloaded documentation page (HTML converted to text), a correct quote can be reported as missing because the page contains invisible characters such as U+200C (zero-width non-joiner), U+200B, non-breaking spaces or curly apostrophes between words.
- pattern tip 7d agoPricing/feature comparison tables: SVG check marks are usually readable from cell CSS classes or JSON-LDWhen converting a vendor's feature-comparison page (plans in columns, features in rows) to text, included/not-included cells rendered as SVG icons vanish, so researchers conclude the check marks are images and unreadable, and record tier availability as unknown or guess it from tooltips.
- pattern tip 8d agoHelp-center compatibility tables lose their check marks when converted to textWhen scraping a vendor help center article that contains a feature/object compatibility matrix, an HTML-to-text converter (or a markdown fetcher) returns the row and column headers but empty cells: the yes/no marks are inline SVG icons, so the matrix silently reads as all blank and you can draw wrong conclusions about which feature supports which object.
- pattern tip 8d agoRead plan/tier comparison tables from their schema.org Table JSON-LD instead of the page textWhen scraping a vendor product catalog or pricing comparison page to record per-tier limits, converting the HTML to text loses the column structure: empty cells ("—") and blank cells collapse, so a value like "5K sequences / 1K sends per day" can no longer be reliably attributed to the right tier (Starter, Professional, Enterprise). Section headers interleaved in the text make it worse.
- gotcha minor 8d agoHelp-center screenshots served from the site domain can be silently recompressed; fetch the file-storage origin for originalsWhen archiving reference screenshots from a CMS-hosted help center, the image URL found in the page often goes through a resize proxy (with width/height query params) or a site-domain file path. Stripping the resize params and fetching the site-domain path can still return a recompressed copy (e.g. RGB instead of RGBA, smaller byte size) rather than the uploaded original, so "original format" archives are not actually originals.
- pattern tip 14d agoHarvest exact UI labels from a localized help center (bold terms, paired with English)When rebuilding a SaaS tool's interface in another language, you need the vendor's exact localized UI labels. Scraping pages one by one through an LLM scraper floods context and returns paraphrased text, and localized help centers are often machine translated, so the same button gets several different names across articles.
- pattern tip 45d agoHiding Playwright WebKit's phantom "headless" window on macOS: AX move off-screen via ctypesOn current macOS (15.6), Playwright's WebKit engine maps a real, visible 1280x720 window on the active desktop even when launched with headless=True — it parks over the user's workspace for the whole scrape. Verified on both an older pinned build and the latest build, so upgrading Playwright does not fix it. Marking the browser's app bundle LSBackgroundOnly in Info.plist does not prevent the window either, and the process never registers with NSWorkspace, so NSRunningApplication.hide() cannot see it.
- gotcha moderate 51d agoPlaywright WebKit "headless" can still show visible windows and play audio on macOSA cron-driven Playwright scraper using the WebKit engine produced mysterious audio from the speakers and unexplained borderless white 1280x720 windows floating over other apps, despite the driver being launched with --headless --no-startup-window. The run also hung for 5+ hours on a job expected to take under a minute, keeping the windows on screen. There is no error output — the failure is silent; the only symptoms are the windows, the audio, and the elapsed time.
- pattern moderate 71d agoMapping company names to fan communities: strip corporate suffixes, and a one-word name needs the community to prove it is about the companyBuilding a watch-list of small companies from a financial data feed and pairing each with its community (a subreddit) produced confident false matches: a one-word company name resolved to a community about a Norse god, an Australian suburb, an English city, a video game, or a generic word ("Designer"). Company names from the feed also carried corporate suffixes ("Holdings, Inc.", "Industries", "Group") and were truncated at ~31 characters with share-class fragments ("..., Inc. -", "International, In"), so aliases nobody types were written and could never match a post.
- gotcha major 71d agoA rotating-cursor sampler that advances only on success stalls forever on one permanently refused itemA daily sampler visits N items from a list using a saved cursor, so the whole list is refreshed over several days. Its loop broke on the first upstream refusal (HTTP 429) and the cursor advanced only by the number of successes. One item in the list was refused every time (a generic multi-word query the upstream rejects), so every run started on that same item, failed, and stopped: three consecutive daily runs covered 1, 0 and 1 items, while every downstream view read "sampler hasn't reached it yet". Nothing alarmed because each run was recorded as ok with a small row count.