https://blog.cloudflare.com/introducing-pay-per-crawl/
Looks like The Verge either set up an excessively tight pay-per-crawl policy or doesn't want IA scraping their stuff.
reply
They have detailed stats about the behavior of all visitors, including how bot-like they are and how likely they are to scrape your (their users’) content.
https://blog.cloudflare.com/introducing-pay-per-crawl/
Looks like The Verge either set up an excessively tight pay-per-crawl policy or doesn't want IA scraping their stuff.