# Robots.txt for LIT.ME Events # https://litme.events # # ---------------------------------------------------------------------------- # Reservation of rights # ---------------------------------------------------------------------------- # Our editorial text, venue descriptions, collective profiles, page structure, # layout, design and underlying code are the copyright of LIT.ME Events. The # venue directory is additionally protected as a compilation and by UK database # right, reflecting the investment made in obtaining and verifying its contents. # # Access to this site is granted subject to our Terms of Use: # https://litme.events/legal/terms-of-use/ # # Those Terms prohibit systematic collection or extraction of this site's # content, and prohibit its use for training or fine-tuning AI models, or for # the automated reproduction or recreation of this site or its design. # # The Content-Signal lines below express those preferences in machine-readable # form, per the Content Signals Policy (https://contentsignals.org/): # # search — building a search index and returning links and short excerpts # ai-input — supplying content to an AI model at answer time (RAG, grounding) # ai-train — training or fine-tuning AI models # # If a content-signal = yes, you may collect content for the corresponding use. # If a content-signal = no, you may not collect content for the corresponding # use. Where no signal is given for a use, permission is neither granted nor # restricted by this file. # # Our position: cite us, do not train on us, do not clone us. # Content-Signal: search=yes, ai-input=yes, ai-train=no # ---------------------------------------------------------------------------- # Content extraction & site-cloning services — disallowed # ---------------------------------------------------------------------------- # Scraper-as-a-service and page-to-code platforms. These exist to lift a page's # markup, styling and copy wholesale, which is exactly the use our Terms forbid. # # NOTE: several tools in this class (headless-browser services, screenshot APIs, # stealth scrapers) send an ordinary Chrome user-agent and cannot be named here. # robots.txt cannot reach them — the Vercel WAF bot-protection ruleset does. User-agent: FirecrawlAgent Disallow: / User-agent: Diffbot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: Timpibot Disallow: / User-agent: Scrapy Disallow: / # ---------------------------------------------------------------------------- # Aggressive SEO / scraping bots — disallowed to reduce CPU load on SSR routes # ---------------------------------------------------------------------------- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Bytespider Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SeekportBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: YandexBot Disallow: / User-agent: Mail.RU_Bot Disallow: / # ---------------------------------------------------------------------------- # AI Crawlers — allowed for referencing, NOT for training. # See /llms.txt for usage guidelines. # ---------------------------------------------------------------------------- # robots.txt group matching is most-specific-wins: a crawler that matches its # own group reads ONLY that group. Content-Signal must therefore be repeated in # every group rather than declared once under User-agent: *. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: ClaudeBot User-agent: Claude-Web User-agent: Claude-User User-agent: Claude-SearchBot User-agent: anthropic-ai Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: PerplexityBot User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: cohere-ai Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: FacebookBot User-agent: meta-externalagent Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # ---------------------------------------------------------------------------- # Search engines — allow crawling with a crawl delay to limit bursts # ---------------------------------------------------------------------------- User-agent: Googlebot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /api/ Disallow: /iris/ Disallow: /auth/ Disallow: /application-success/ User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /api/ Disallow: /iris/ Disallow: /auth/ Disallow: /application-success/ Crawl-delay: 10 User-agent: DuckDuckBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # ---------------------------------------------------------------------------- # Default — allow, but disallow high-cost / private paths # ---------------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /api/ Disallow: /iris/ Disallow: /auth/ Disallow: /application-success/ Crawl-delay: 10 Sitemap: https://litme.events/sitemap.xml