安裝fake-useragent模塊
~~~
pip install fake-useragent
~~~
在middlewares.py文件中導入模塊
~~~
from fake_useragent import UserAgent
~~~
定義UAMiddleware中間件
~~~
class UAMiddleware(object):
def __init__(self, crawler):
super(UAMiddleware, self).__init__()
self.ua = UserAgent()
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def process_request(self, request, spider):
request.headers.setdefault('User-Agent', self.ua.random)
~~~
在settings.py配置文件DOWNLOADER_MIDDLEWARES中添加該中間件
