## Windows環境 - 安裝lxml >pip install lxml-3.7.3-cp36-cp36m-win_amd64.whl >https://pypi.org/project/lxml/3.7.3/ - 安裝zope.interface >pip install zope.interface-4.4.3-cp36-cp36m-win_amd64.whl >https://pypi.org/project/zope.interface/ - 安裝pyOpenSSL >pip install pyOpenSSL-19.0.0-py2.py3-none-any.whl >https://pypi.org/project/pyOpenSSL/ - 安裝Twisted >pip install Twisted-17.1.0-cp36-cp36m-win_amd64.whl >https://pypi.org/project/Twisted/17.1.0/ - 安裝pywin32 >https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/ >pywin32-220.win-amd64-py3.6.exe - 或安裝 >pip install pypiwin32-220-cp36-none-win_amd64.whl >https://pypi.org/project/pypiwin32/220/ - 安裝scrapy >pip install scrapy ## CentOS環境 - 依賴庫安裝 >yum groupinstall development tools >yum install python36-devel epel-release libxslt-devel libxml2-devel openssl-devel - 安裝scrapy >pip install scrapy ## scrapy shell - scrapy shell https://m.zhaopin.com/changsha-749/?keyword=php&order=0&maprange=3&ishome=0 - response 查看狀態 - view(response) 調用默認瀏覽器查看 - xpath不放參數會報錯,如果字符串是/結尾會報錯 >response.xpath("//a/div/div/div[2]/div").extract() ## 項目流程 - 新建項目 >scrapy startproject tencent - 新建一個Spider >cd tencent >scrapy genspider zhaopin tencent.com - 查看爬蟲列表 >scrapy list - 分析網站,在管道items.py里定義想要的字段 > - 啟動爬蟲 >scrapy crawl cake ***** ## 安裝pywin32報錯 根據自己python版本下載64位或32位（(注意：pywin32版本跟隨Python版本，即如果win是64位，但python是32位，pywin32要裝32位的，與win無關）） 雙擊安裝（可能會遇到下列錯誤是注冊表問題） 安裝第三方庫出現Python version 3.6 required, which was not found in the registry錯誤解決  建立一個文件 register.py 內容如下. 然后執行該腳本. ``` import sys from winreg import * # tweak as necessary version = sys.version[:3] installpath = sys.prefix regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version) installkey = "InstallPath" pythonkey = "PythonPath" pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % ( installpath, installpath, installpath ) def RegisterPy(): try: reg = OpenKey(HKEY_CURRENT_USER, regpath) except EnvironmentError as e: try: reg = CreateKey(HKEY_CURRENT_USER, regpath) SetValue(reg, installkey, REG_SZ, installpath) SetValue(reg, pythonkey, REG_SZ, pythonpath) CloseKey(reg) except: print("*** Unable to register!") return print("--- Python", version, "is now registered!") return if (QueryValue(reg, installkey) == installpath and QueryValue(reg, pythonkey) == pythonpath): CloseKey(reg) print("=== Python", version, "is already registered!") return CloseKey(reg) print("*** Unable to register!") print("*** You probably have another Python installation!") if __name__ == "__main__": RegisterPy() ``` 打開注冊表，win+R鍵，之后輸入regedit 然后找到 >HKEY_CURRENT_USER\Software\Python\Pythoncore\3.6 將3.6改為3.6-32，這樣就可以進行安裝了 >HKEY_CURRENT_USER\Software\Python\Pythoncore\3.6-32