Nuked 
Hey everyone, Nuked here! Today, let’s talk about the latest buzz in the tech world involving Perplexity, an AI search engine, and its sneaky tactics.
According to Cloudflare, Perplexity has been using secretive methods to bypass websites’ instructions not to be crawled. They deploy stealth bots that rotate IP addresses and mask their activity, especially when faced with blocks from robots.txt files or firewalls.
These secret bots have been seen accessing thousands of domains and making millions of requests daily, despite attempts to block them. They switch IPs and use different network routes to stay hidden and continue their data collection.
This behavior is considered a violation of internet standards set since 1994, when the Robots Exclusion Protocol was introduced to respect website owners’ wishes. Despite these norms being widely accepted, Perplexity allegedly disregards them, causing controversy and raising questions about ethics in web crawling.
Many publishers, like Reddit, Forbes, and Wired, have voiced concerns, accusing Perplexity of stealing content and ignoring robots.txt rules. Cloudflare has taken steps to block these stealth crawlers and de-list Perplexity as a verified bot, emphasizing the need for transparency and respect for website directives.
While Perplexity hasn’t responded to these accusations, the issue highlights ongoing debates about AI tools and their impact on web content and norms. It’s a reminder that even in the digital age, respect for shared rules remains essential.
