# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # --------------------------------------------------------------------------- # mcpjunction.ai — crawler policy # # The Content Signals Policy above is reproduced verbatim so it travels with # this file wherever it is served. It is NOT injected by Cloudflare here: # Cloudflare's "managed robots.txt" is deliberately DISABLED for this zone, # because when enabled it REPLACED this file rather than prepending to it, # silently dropping the Sitemap and License directives below. Enforcement # still runs through AI Crawl Control, which is a separate feature. # # If you ever re-enable managed robots.txt, re-verify that /robots.txt still # contains the Sitemap: and License: lines at the bottom. If it does not, # turn it back off — those two lines are load-bearing. # # Posture: search welcome, agents welcome during the authority-building phase # (we want citations), training crawlers licensed rather than free. # Revisit quarterly. # --------------------------------------------------------------------------- # --- Default group. Content signals live here. --- # ai-input=yes is the launch-phase posture: we WANT agents grounding answers # in this data and citing us. Flip to ai-input=no + Pay Per Crawl when bot # analytics show several thousand agent hits/month. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Crawl-delay: 2 # Content-Signal is repeated in every group below, not just the default one. # robots.txt group semantics mean a crawler obeys ONLY its most specific # matching group: a bot with its own group never reads the `*` group, and so # would never see the signal there. Since these signals are an express # reservation of rights under Article 4 of EU 2019/790, they have to travel # with the group that each crawler actually reads. # --- Search crawlers: always welcome (they send humans) --- User-agent: Googlebot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: DuckDuckBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # --- AI agents acting for a user: ALLOWED (citation strategy) --- User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # Meta's USER-INITIATED fetcher — the counterpart to ChatGPT-User, not to be # confused with meta-externalagent below, which is Meta's training crawler and # stays blocked. Cloudflare classes this one as an AI Assistant. User-agent: Meta-ExternalFetcher Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # Answer-engine indexers. These build a searchable index used to generate # cited answers — the same job Googlebot does for blue links, which is why # they sit with the allowed group rather than the training crawlers. Being # in these indexes is the point of the whole citation strategy. User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # --- Training crawlers: licensed, not free. Terms in license.xml. # Enforcement layer = Cloudflare AI Crawl Control (Training: block). --- User-agent: GPTBot Content-Signal: ai-train=no Disallow: / User-agent: ClaudeBot Content-Signal: ai-train=no Disallow: / User-agent: anthropic-ai Content-Signal: ai-train=no Disallow: / User-agent: Google-Extended Content-Signal: ai-train=no Disallow: / User-agent: CCBot Content-Signal: ai-train=no Disallow: / User-agent: Applebot-Extended Content-Signal: ai-train=no Disallow: / User-agent: Bytespider Content-Signal: ai-train=no Disallow: / User-agent: meta-externalagent Content-Signal: ai-train=no Disallow: / User-agent: Amazonbot Content-Signal: ai-train=no Disallow: / User-agent: AI2Bot Content-Signal: ai-train=no Disallow: / User-agent: Diffbot Content-Signal: ai-train=no Disallow: / User-agent: cohere-ai Content-Signal: ai-train=no Disallow: / User-agent: Omgilibot Content-Signal: ai-train=no Disallow: / User-agent: ImagesiftBot Content-Signal: ai-train=no Disallow: / User-agent: YouBot Content-Signal: ai-train=no Disallow: / User-agent: Timpibot Content-Signal: ai-train=no Disallow: / User-agent: PanguBot Content-Signal: ai-train=no Disallow: / Sitemap: https://mcpjunction.ai/sitemap.xml # --- RSL (Really Simple Licensing) --- License: https://mcpjunction.ai/license.xml