User-agent: * Allow: / # Principle for this file: only Disallow what should never be *fetched*. # Everything else that should stay out of the index says so with "noindex" via # useSeo(). The two are not interchangeable: a disallowed URL is never crawled, # so its noindex is never read, and an already-linked URL can still surface as a # bare result. Disallow also blocks link-preview bots (LinkedIn, Slack, X). # Product surface, behind auth. Nothing here should be fetched at all. Disallow: /app Disallow: /admin # Tokenised unsubscribe/preference links arrive by email and carry a ?token. # Bots have no business fetching them. Disallow: /email-preferences # Deliberately NOT disallowed, all of them rely on "noindex, follow" instead: # /trial outreach landing page. Blocking it would kill the link # preview card when the URL is shared on LinkedIn or Slack, # which is the whole point of the page. See the comment in # src/pages/Trial.tsx, which documents this decision. # /ad/ ad detail pages, noindex today. Revisit when they get an # indexing strategy. # /login, /forgot-password, /reset-password # already noindex; Googlebot renders and reads it. Blocking # would trade guaranteed de-indexing for a crawl block. Sitemap: https://adpeekly.com/sitemap.xml