**一、Apache** **①、通過修改 .htaccess 文件** 修改網站目錄下的.htaccess，添加如下代碼即可(2 種代碼任選)： 可用代碼 (1)： ``` RewriteEngine?On RewriteCond?%{HTTP\_USER\_AGENT}?(^$|FeedDemon|Indy?Library|Alexa?Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft?URL?Control|Swiftbot|ZmEu|oBot|jaunty|Python–urllib|lightDeckReports?Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)?\[NC\]? RewriteRule?^(.\*)$?–?\[F\] ``` 可用代碼 (2)： ``` SetEnvIfNoCase?^User–Agent$?.\*(FeedDemon|Indy?Library|Alexa?Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft?URL?Control|Swiftbot|ZmEu|oBot|jaunty|Python–urllib|lightDeckReports?Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)?BADBOT? Order?Allow,Deny? Allow?fromall Deny?from?env=BADBOT ``` **②、通過修改 httpd.conf 配置文件** 找到如下類似位置，根據以下代碼 新增 / 修改，然后重啟 Apache 即可： ``` DocumentRoot?/home/wwwroot/xxx? SetEnvIfNoCase?User–Agent?“.\*(FeedDemon|Indy?Library|Alexa?Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft?URL?Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports?Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms)”?BADBOT? ????????Order?allow,deny? ????????Allow?fromall ???????deny?from?env=BADBOT? ``` **二、Nginx 代碼** 進入到 nginx 安裝目錄下的 conf 目錄，將如下代碼保存為 agent\_deny.conf ``` cd?/usr/local/nginx/conf? vim?agent\_deny.conf ``` ``` #禁止Scrapy等工具的抓取? if?($http\_user\_agent?~\*?(Scrapy|Curl|HttpClient))?{? ?????return?403;? }? #禁止指定UA及UA為空的訪問? if?($http\_user\_agent?~\*?“FeedDemon|Indy?Library|Alexa?Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft?URL?Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports?Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|^$”?)?{? ????return?403;????????????? }? #禁止非GET|HEAD|POST方式的抓取? if?($request\_method?!~?^(GET|HEAD|POST)$)?{? ????return?403;? } ``` 然后，在網站相關配置中的 location / { 之后插入如下代碼： **Shell** ``` include?agent\_deny.conf;? ``` 如下的配置： **Shell** ``` \[marsge@Mars\_Server?~\]$?cat?/usr/local/nginx/conf/zhangge.conf? location?/?{? ????????try\_files?$uri?$uri/?/index.php?$args;? ????????#這個位置新增1行：? ????????include?agent\_deny.conf;? ????????rewrite?^/sitemap\_360\_sp.txt$?/sitemap\_360\_sp.php?last;? . ????????rewrite?^/sitemap\_baidu\_sp.xml$?/sitemap\_baidu\_sp.php?last;? . ????????rewrite?^/sitemap\_m.xml$?/sitemap\_m.php?last;? ``` 保存后，執行如下命令，平滑重啟 nginx 即可： Shell ``` /usr/local/nginx/sbin/nginx?–s?reload ``` **三、PHP 代碼** 將如下方法放到貼到網站入口文件 index.php 中的第一個 **PHP** ``` //獲取UA信息? $ua?=?$\_SERVER\[‘HTTP\_USER\_AGENT’\];? //將惡意USER\_AGENT存入數組? $now\_ua?=?array(‘FeedDemon?‘,‘BOT/0.1?(BOT?for?JCE)’,‘CrawlDaddy?‘,‘Java’,‘Feedly’,‘UniversalFeedParser’,‘ApacheBench’,‘Swiftbot’,‘ZmEu’,‘Indy?Library’,‘oBot’,‘jaunty’,‘YandexBot’,‘AhrefsBot’,‘MJ12bot’,‘WinHttp’,‘EasouSpider’,‘HttpClient’,‘Microsoft?URL?Control’,‘YYSpider’,‘jaunty’,‘Python-urllib’,‘lightDeckReports?Bot’);? //禁止空USER\_AGENT，dedecms等主流采集程序都是空USER\_AGENT，部分sql注入工具也是空USER\_AGENT? if(!$ua)?{? ????header(“Content-type:?text/html;?charset=utf-8”);? ????die(‘請勿采集本站，因為采集的站長木有小JJ！’);? }else{? ????foreach($now\_ua?as?$value?)? //判斷是否是數組中存在的UA? ????if(eregi($value,$ua))?{? ????????header(“Content-type:?text/html;?charset=utf-8”);? ????????die(‘請勿采集本站，因為采集的站長木有小JJ！’);? ????}? } ``` **四、測試效果** 模擬宜搜蜘蛛抓取： **Shell** ``` curl?–I?–A?‘YisouSpider’?bizhi.bcoderss.com? ``` 模擬 UA 為空的抓取： **Shell** ``` curl?–I?–A?”?bizhi.bcoderss.com? ``` 模擬百度蜘蛛的抓取： **Shell** ``` curl?–I?–A?‘Baiduspider’?bizhi.bcoderss.com ```