Вопрос Настройка Robots.txt

[Optinice]

Привет форумчане, подскажите на сколько правильный у меня robots.txt
Может стоить что то убрать или дополнить ?

Спойлер: robots.txt

User-agent: *
Disallow: /admin.php
Disallow: /account/
Disallow: /attachments/
Disallow: /search/
Disallow: /lfs/
Disallow: /goto/
Disallow: /misc/
Disallow: /birthdays/
Disallow: /threads/*/reply
Disallow: /*/about
Disallow: /*/unread
Disallow: /*/latest
Disallow: /threads/*/who-replied/
Disallow: /conversations/
Disallow: /lost-password/
Disallow: /online/
Disallow: /misc/style?*
Disallow: /cdn-cgi/
Disallow: /*/page-*
Disallow: /*?
Sitemap: https://domen/sitemap.xml

User-agent: Yandex
Clean-param: content&user_id&prefix_id&desc&page&download_count&direction&reject&accept&update&_debug&direction&order&tab_id
Disallow: /whats-new/
Disallow: /members/
Disallow: /resources/categories/*/featured
Disallow: /resources/*/update/*/reactions
Disallow: /resources/*/updates
Disallow: /resources/*/reviews
Disallow: /resources/*/history
Disallow: /resources/authors/*/
Disallow: /help/
Disallow: /login/
Disallow: /register/
Disallow: /tags/

User-agent: YandexNews
Allow: /forums/-/index.rss

User-agent: Googlebot-News
Allow: /forums/-/index.rss

User-agent: Mail.Ru
Crawl-delay: 2

User-agent: StackRambler
Crawl-delay: 2

User-agent: Baiduspider
Crawl-delay: 5

User-agent: YaCy
Crawl-delay: 10

User-agent: Phind
Crawl-delay: 3

User-agent: Wolfram
Crawl-delay: 5

User-agent: Waldo
Crawl-delay: 2

User-agent: Ask
Crawl-delay: 2

User-agent: bingbot
Crawl-delay: 2

User-agent: DuckDuckBot
Crawl-delay: 1

User-agent: Slurp
Crawl-delay: 2

User-agent: Yeti
Crawl-delay: 3

User-agent: Applebot
Crawl-delay: 1

User-agent: StartPage
Crawl-delay: 2

User-agent: PerplexityBot
Crawl-delay: 3

User-agent: YouBot
Crawl-delay: 2

User-agent: FindSounds
Crawl-delay: 2

User-agent: TinEye
Crawl-delay: 3

User-agent: Mediapartners-Google
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: trendictionbot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: MegaIndex
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-Embedder
Disallow: /

User-agent: OAI-SearchCrawler
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: MJ12bot
Crawl-delay: 10
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 5
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: adbeat_bot
Disallow: /

User-agent: seobots
Disallow: /

User-agent: spbot
Disallow: /

 

Вопрос  Продвижение форума

Вопрос  Узнать больше об этом ресурсе в виде кнопки