robots规则以及阻止垃圾爬虫

   
摘要GPT
摘要小助理暂时失联跑路啦……😜
User-agent: *
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /go/
Disallow: /wp-content/plugins/
Disallow: /?s=*
Disallow: /author/
Disallow: /astra-advanced-hook/
Disallow: /theme
Disallow: /demo/*
Disallow: /*/page/*
Disallow: /*blackhole
Disallow: /?blackhole
Allow: /wp-admin/admin-ajax.php

User-agent: YandexBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: YaK
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: MJ12bot
Disallow: /

Sitemap: https://www.xxx.com/sitemap.xml
知识共享许可证 CC BY-NC-SA 4.0
最后更新于 2024-06-14 02:32
使用 Hugo 构建
主题 hugo-magic小洋葱 魔改 由 Jimmy 设计
Written by Human, Not by AI