I'm Jed, the author. I wrote this guide up because I'm working on an AI search startup using agents, and have been asked by other founders a few times why their content doesn't show up well with AI search engines like ours or Perplexity or ChatGPT with search, or with agent computer-use like the new OpenAI Operator.
We're ingesting around 30m to 50m pages a day so have learned a lot about what blocks AI crawlers and agents. And I wanted to share some guidelines on how to make sure your startup or documentation or other content shows up as well as possible for agent use.
We're ingesting around 30m to 50m pages a day so have learned a lot about what blocks AI crawlers and agents. And I wanted to share some guidelines on how to make sure your startup or documentation or other content shows up as well as possible for agent use.