User-agent: * Allow: / # Company detail pages and industry-filtered explore views are the pages worth # crawling. Other query-parameter combinations on /product are near-duplicates # of those views, so they are kept out of the crawl budget. Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= # Account and intake pages have nothing to index. Disallow: /login Disallow: /signup # ─── AI answer engines ─── # Same rule as above: real company pages and industry views are open, redundant # query-parameter variations of the search page are not. User-agent: GPTBot Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: ChatGPT-User Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: OAI-SearchBot Allow: / Disallow: /login Disallow: /signup User-agent: ClaudeBot Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: Claude-Web Allow: / Disallow: /login Disallow: /signup User-agent: anthropic-ai Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: Google-Extended Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: PerplexityBot Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup User-agent: CCBot Allow: / Disallow: /product?*q= Disallow: /product?*country= Disallow: /product?*size= Disallow: /product?*sort= Disallow: /product?*page= Disallow: /product?*view= Disallow: /product?*language= Disallow: /product?*window= Disallow: /product?*ranked= Disallow: /product?*ai= Disallow: /login Disallow: /signup Sitemap: https://customereye.ai/sitemap.xml RSS: https://customereye.ai/rss.xml