# robots.txt for meiri.co
# עו״ד ורו״ח ירון מאירי — Yaron Meiri, Adv. & CPA
# Search engines and AI crawlers are welcome; the bot policy is below.
# Machine-readable summary:  https://meiri.co/llms.txt
# Full text of core pages:   https://meiri.co/llms-full.txt
#
# Bot policy (reviewed 30.9.2026, a conscious decision per bot):
# - One group for all search engines and AI crawlers, so every one of them gets
#   the same Disallow lines. A crawler obeys only the most specific group that
#   names it, so separate "Allow: /" groups would drop those lines for that bot.
# - Bytespider is the only crawler that is blocked.
# - 26.10.2026: Google-CloudVertexBot, DuckAssistBot, meta-externalfetcher, YouBot and Timpibot named in the
#   same group (same rules). Brave Search has no token of its own and follows the Googlebot rules.
# - Disallow lines come before "Allow: /" so that first-match parsers read them
#   the same way as longest-match parsers (Google, Bing, RFC 9309).

User-agent: *
User-agent: Googlebot
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: DuckDuckBot
User-agent: DuckAssistBot
User-agent: YandexBot
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: MistralAI-User
User-agent: YouBot
User-agent: Timpibot
Disallow: /agent/
Disallow: /netlify/
Disallow: /_build/
Allow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://meiri.co/sitemap.xml