Web scraping for massive amounts of data can arguably be described as the secret sauce of generative AI. After all, AI chatbots like ChatGPT, Claude, Bard and LLaMA can spit out coherent text because they were trained on massive corpora of data, mostly scraped from the internet. And as the size of today’s LLMs like GPT-4 have ballooned to hundreds of billions of tokens, so has the hunger for…
