## 在linux下安裝并運行scrapyd
```
pip isntall scrapyd
mkdir /etc/scrapyd
vi /etc/scrapyd/scrapyd.conf
```
```
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 5
dbs_dir = dbs
max_proc = 0
max_proc_per_cpu = 10
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
```
>制作軟連接: ln -s /usr/local/python3/bin/scrapyd /usr/bin/scrapyd
>創建文件后執行命令啟動scrapyd: (scrapyd > /dev/null &) 當想要記錄輸出日志時: (scrapyd > /root/scrapyd.log &)
- 工欲善其事必先利其器
- 請求庫
- 解析庫
- 數據庫
- 存儲庫
- Web庫
- app爬取相關庫
- 爬蟲框架
- 部署相關庫
- ipython
- 基礎
- 數學函數
- 隨機函數
- 三角函數
- 字符串內建函數
- 列表方法
- 字典內置方法
- 正則表達式
- os
- 字符串及數字的判斷
- 常用魔術方法
- db
- mongodb
- mysql
- redis
- ORM
- ODM
- mongodb操作方法
- sqlite3
- access
- files
- Excel
- xml文件
- Python環境
- anaconda
- pip常用命令
- virtualenv
- pyenv
- cmder
- 遠程開發
- Jupyter
- crawler
- appium環境搭建
- adb工具
- uiautomator
- 運行Appium+Python Clinet + 夜神模擬器
- DesiredCapabilities參數大全
- requests
- scrapy
- gerapy
- scrapyd
- 請求頭fake_useragent庫
- 數據傳遞過程
- 數據清洗及入庫pipelines.py
- scrapy調用阿布云代理
- 圖片下載
- PyQt5
- pyinstaller
- 攻防
- xss
- xss反射
- Chrome模擬微信瀏覽器
- flask
- 注冊app
- 藍圖Blueprint
- 表單驗證wtforms
- Flask-SQLAlchemy
- 數據處理
- json
- tornado
- settings
- 工具
- fiddler
- ab壓力測試工具
- 高階
- 隊列
- 多線程
- 消息隊列
- 定時任務框架APScheduler
- Django
- 路由分離
- 模型
- admin
- Android
- apk逆向工程