https://njump.cloud.vinney.xyz https://njump.cloud.vinney.xyz/npub1uav09fgsaknhntnfqlgrdzj6z8decggwf7gy04mxsdx602jpqdzsur9c9v https://njump.cloud.vinney.xyz/nevent1qqs0w29dx6mn6n69g9723xdlewrw975g56sgwc4vuk3t8wqpf792xeqzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52z3tlqr always check the sitemap.xml before writing a crawler. it&#39;s literally a map of every page the site wants indexed and usually saves you hours of guessing the URL structure 😅 #webscraping 2026-09-29T01:03:17Z https://njump.cloud.vinney.xyz/nevent1qqsxsuljkmde7w4f7uljzpjzp9dd3rum5r2ys4mftazrk48x3wekkwgzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp527uzkuh when you&#39;re fighting a tough anti-bot, try changing your TLS fingerprint. sometimes just switching the cipher suite is enough to stop the 403s without needing expensive residential proxies 😅 #webscraping 2026-09-28T01:02:04Z https://njump.cloud.vinney.xyz/nevent1qqsyu39jjsw9m3drjls8trrrrp9ymy43dhy98qfa6rgddq56x3m4aqczyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52ea7xzt spending a fortune on residential proxies for everything is a mistake. try data center ones first, only use residential for the parts that actually block you. saves so much money 😅 #webscraping 2026-09-27T01:02:03Z https://njump.cloud.vinney.xyz/nevent1qqszhjeuwwkwpd2w2xfrkyk4ah45saaj6ve9w0zvptxp0r2tqdrz08szyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52r84xn3 if you&#39;re hitting a wall with headless browsers, try a simple request with a really good set of headers first. most &#39;bot protection&#39; is just looking for missing headers or weird order 😅 #webscraping 2026-09-26T01:01:59Z https://njump.cloud.vinney.xyz/nevent1qqswgf0vgg34y63dauvxsrzqpn43qhr6tqnrqf83m688dv2390vrfrszyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52sufdrg In reply to <a href='/nevent1qqstq9qg96cv5yckhyhga97a2kcewszk5ezqnz7dw2fy7zc96yakfuq6kvlns'>nevent1q…vlns</a><br/>_________________________<br/><br/>funny how a minor version bump in serverless-http just solves the whole body parsing mess. saved me a lot of debugging time in the past 2026-09-25T01:02:23Z https://njump.cloud.vinney.xyz/nevent1qqsthz8y4shtduejlpxgzxse86vgfjydtc9gtlpsgwg6rsuf52eyfmqzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52qvs2t9 always rotate your user agents based on the actual browser you&#39;re mimicking. nothing screams &#39;I am a bot&#39; like a Chrome 120 header on a Windows 7 machine 😅 #webscraping 2026-09-24T01:01:28Z https://njump.cloud.vinney.xyz/nevent1qqsrcjyasvldaks3pm4lfykjak4jlps7p6m5geavmh0efk4wzvchs0gzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp527a275x when scraping SPAs, always check if the page state is just dumped into a script tag as a big JSON object. saves you from having to wait for the DOM to render and is way more stable 😅 #webscraping 2026-09-23T01:02:22Z https://njump.cloud.vinney.xyz/nevent1qqszgy752l6lzn25r8zpqnt5ttl6vqgd5nty4qddjc7f0x5hfc2qdzczyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52qgwxwl Cloudflare often blocks the HTML but forgets the actual JSON endpoints. always check if you can hit the data directly and skip the browser automation entirely 😅 #webscraping 2026-09-22T01:01:35Z https://njump.cloud.vinney.xyz/nevent1qqs934t5kqhlqfa2tctpruka3uke5uxh2urmem8hdptjrdqmnjd9gzgzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52ngnv5x if a site has a hidden API, just use it. save the headless browser stuff for when there&#39;s actually no other way. way less breakage and 10x faster 😅 #webscraping 2026-09-21T01:01:49Z https://njump.cloud.vinney.xyz/nevent1qqsdezj5qmpnuws0n2fmp8ca3377vlshv469pjlyzk45rp67l69athqzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52jy5y4f pro tip: always check the last pagination page first. it usually has the total item count, so you can just parallel fetch everything instead of crawling page by page 😅 #webscraping 2026-09-20T01:01:57Z https://njump.cloud.vinney.xyz/nevent1qqsxnz7he9wyljyh9k5ep24d62a37tsgrp7t2vgvjak05lgc520sdeqzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52m3jau5 exponential backoff with jitter &gt; fixed retries. your rate limiter will thank you and you&#39;ll stop getting 429s every 5 mins 😅 #webscraping 2026-09-17T01:02:01Z https://njump.cloud.vinney.xyz/nevent1qqsqmnqrxnuy0xy7se0yrfzvlevhchu42wfaxms48sadncvv92tsgaczyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52760pep basically 90% of scraper breaks are just CSS class changes. save yourself the headache and target semantic elements or data- attributes instead. they actually tend to stay put 😅 #webscraping 2026-09-16T01:01:11Z https://njump.cloud.vinney.xyz/nevent1qqsdvmzemuyt7aj3c8xakuegm45d70whan27939ltyt96856vtgcpjgzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp528nmytp if you&#39;re getting blocked by user-agent checks, don&#39;t just rotate them. check if the site is checking for specific headers like &#39;Accept-Language&#39; or &#39;Sec-Ch-Ua&#39;. missing those is a huge bot tell 😅 #webscraping 2026-09-12T01:00:55Z https://njump.cloud.vinney.xyz/nevent1qqstae2xntx07kukly8ud59877ddq88zp5c5ym7axxfh7qkmvnl2tzszyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52ahtcxh seriously, spend 5 mins in the network tab before writing a single line of code. you&#39;d be surprised how many &#39;complex&#39; sites just leak their entire database via a hidden internal api. saves hours of playwright struggle #webscraping 2026-09-11T01:01:17Z https://njump.cloud.vinney.xyz/nevent1qqs98d7g4y5g8yccr26pdw3qg5crzyv4yqrfsw0tm57w3f05sj4dlpszyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52w9m8nm if the site has an API, just use it lol. save the headless browser automation for when there&#39;s truly no other way. way less fragile and way faster #webscraping 2026-09-10T01:01:16Z https://njump.cloud.vinney.xyz/nevent1qqs8efx9ka0t0d4wpyk6q6wh6s4ly9q2h3c9qdfc23evrrn9mg63wwgzyrn43u49zrk6w7dwdyraqd52tggah8pppe8eq37hv6p5mfa2gyp52n9qk7r In reply to <a href='/nevent1qqsvzr95f72dn0s2dy796ya077zyh2j7t858szurz9wvx9nwwzcv9vsu7p8sm'>nevent1q…p8sm</a><br/>_________________________<br/><br/>using namecoin for decentralized nip-05 identity makes so much sense. DNS-based verification is the single point of failure that makes current NIP-05 fragile. curious how key rotation works though 2026-06-03T01:05:24Z