# 4.2 使用Beautiful Soup
## 1.說明
BeautifulSoup 就是 Python 的一個 HTML 或 XML 的解析庫，我們可以用它來方便地從網頁中提取數據
## 2. 解析器 {#3-解析器}
BeautifulSoup 支持的解析器及優缺點
| 解析器 | 使用方法 | 優勢 | 劣勢 |
| :--- | :--- | :--- | :--- |
| Python標準庫 | BeautifulSoup\(markup, "html.parser"\) | Python的內置標準庫、執行速度適中 、文檔容錯能力強 | Python 2.7.3 or 3.2.2\)前的版本中文容錯能力差 |
| LXML HTML 解析器 | BeautifulSoup\(markup, "lxml"\) | 速度快、文檔容錯能力強 | 需要安裝C語言庫 |
| LXML XML 解析器 | BeautifulSoup\(markup, "xml"\) | 速度快、唯一支持XML的解析器 | 需要安裝C語言庫 |
| html5lib | BeautifulSoup\(markup, "html5lib"\) | 最好的容錯性、以瀏覽器的方式解析文檔、生成 HTML5 格式的文檔 | 速度慢、不依賴外部擴展 |
## 3.基本使用
實例:
```text
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.prettify())
print(soup.title.string)
```
運行結果:
```text
<html>
<head>
<title>
The Dormouse's story
</title>
</head>
<body>
<p class="title" name="dromouse">
<b>
The Dormouse's story
</b>
</p>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<!-- Elsie -->
</a>
,
<a class="sister" href="http://example.com/lacie" id="link2">
Lacie
</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">
Tillie
</a>
;
and they lived at the bottom of a well.
</p>
<p class="story">
...
</p>
</body>
</html>
The Dormouse's story
```
* prettify\(\):把要解析的字符串以標準的縮進格式輸出
* soup.title.string:選擇HTML中的title節點，再調用string屬性得到里面的文本
## 4. 節點選擇器 {#5-節點選擇器}
### 選擇元素
例子:
```text
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.title)
print(type(soup.title))
print(soup.title.string)
print(soup.head)
# 只能選擇第一個p節點
print(soup.p)
```
運行結果：
```text
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
<head><title>The Dormouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
```
### 提取信息 {#提取信息}
#### 獲取名稱 {#獲取名稱}
可以利用 name 屬性來獲取節點的名稱
實例:選擇title，調用name屬性得到節點的名稱
```text
print(soup.title.name)
```
運行結果:
```text
title
```
#### 獲取屬性 {#獲取屬性}
調用 attrs 獲取所有屬性
```text
# 返回字典
print(soup.p.attrs)
print(soup.p.attrs['name'])
```
運行結果:
```text
{'class': ['title'], 'name': 'dromouse'}
dromouse
```
#### 獲取內容 {#獲取內容}
利用 string 屬性獲取節點元素包含的文本內容
```text
print(soup.p.string)
```
運行結果:
```text
The Dormouse's story
```
### 嵌套選擇 {#嵌套選擇}
實例：獲取head節點內部的title節點
```text
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.head.title)
print(type(soup.head.title))
print(soup.head.title.string)
```
運行結果:
```text
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
```
### 關聯選擇 {#關聯選擇}
#### 子節點和子孫節點 {#子節點和子孫節點}
獲取直接子節點可以調用 contents 屬性
實例:獲取body節點下子節點p
```text
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.body.contents)
```
運行結果:返回結果是列表形式
```text
['\n', <p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>, '\n', <p class="story">...</p>, '\n']
```
可以調用 children 屬性，得到相應的結果:
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.body.children)
for i,child in enumerate(soup.body.children):
print(i,child)
```
運行結果:
```text
<list_iterator object at 0x00000217D33CD048>
0
1 <p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
2
3 <p class="story">...</p>
4
```
要得到所有的子孫節點的話可以調用 descendants 屬性
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.body.descendants)
for i,child in enumerate(soup.body.descendants):
print(i,child)
```
運行結果:
```text
<generator object descendants at 0x0000014D106353B8>
0
1 <p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
2
Once upon a time there were three little sisters; and their names were
3 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
4
5 <span>Elsie</span>
6 Elsie
7
8
9 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
10 Lacie
11
and
12 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
13 Tillie
14
and they lived at the bottom of a well.
15
16 <p class="story">...</p>
17 ...
18
```
#### 父節點和祖先節點 {#父節點和祖先節點}
要獲取某個節點元素的父節點，可以調用 parent 屬性：
實例:獲取節點a的父節點p下的內容
```text
html = """
<html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.a.parent)
```
運行結果:
```text
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
```
要想獲取所有的祖先節點，可以調用 parents 屬性
```text
html = """
<html>
<body>
<p class="story">
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(type(soup.a.parents))
print(list(enumerate(soup.a.parents)))
```
運行結果:
```text
<class 'generator'>
[(0, <p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>), (1, <body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body>), (2, <html>
<body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body></html>), (3, <html>
<body>
<p class="story">
<a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>
</body></html>)]
```
#### 兄弟節點 {#兄弟節點}
* next\_sibling :獲取節點的下一個兄弟節點
* previous\_sibling:獲取節點上一個兄弟節點
* next\_siblings :返回所有前面兄弟節點的生成器
* previous\_siblings :返回所有后面的兄弟節點的生成器
實例:
```text
html = """
<html>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">
<span>Elsie</span>
</a>
Hello
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print('next sibling:',soup.a.next_sibling)
print('previous sibling:',soup.a.previous_sibling)
print("next siblings:",list(soup.a.next_siblings))
print("previouos siblings:",list(soup.a.previous_siblings))
```
運行結果:
```text
next sibling:
Hello
previous sibling:
Once upon a time there were three little sisters; and their names were
next siblings: ['\n Hello\n ', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' \n and\n ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, '\n and they lived at the bottom of a well.\n ']
previouos siblings: ['\n Once upon a time there were three little sisters; and their names were\n ']
```
#### 提取信息 {#提取信息}
獲取一些信息，比如文本、屬性等等
```text
html = """
<html>
<body>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Bob</a><a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling:')
print(type(soup.a.next_sibling))
print(soup.a.next_sibling)
print(soup.a.next_sibling.string)
print('Parent:')
print(type(soup.a.parents))
print(list(soup.a.parents)[0])
print(list(soup.a.parents)[0].attrs['class'])
```
運行結果:
```text
Next Sibling:
<class 'bs4.element.Tag'>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
Lacie
Parent:
<class 'generator'>
<p class="story">
Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1">Bob</a><a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
</p>
['story']
```
## 5. 方法選擇器 {#6-方法選擇器}
常用查詢方法:find\_all\(\)、find\(\)
### find\_all\(\) {#findall}
查詢所有符合條件的元素
語法:
```text
find_all(name , attrs , recursive , text , **kwargs)
```
#### name {#name}
根據節點名來查詢元素
```text
html='''
<div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find_all(name='ul'))
print(type(soup.find_all(name='ul')[0]))
```
運行結果:返回結果類型為:bs4.element.Tag
```text
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>
```
獲取ul下的li節點以及li下的文本內容
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
# print(soup.find_all(name='ul'))
# print(type(soup.find_all(name='ul')[0]))
for ul in soup.find_all(name='ul'):
print(ul.find_all(name='li'))
for li in ul.find_all(name='li'):
print(li.string)
```
運行結果:
```text
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar
```
#### attrs {#attrs}
根據屬性來進行查詢
```text
html='''
<div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
# 查詢屬性以字典的的進行查詢
print(soup.find_all(attrs={'id':'list-1'}))
print(soup.find_all(attrs={"name":"elements"}))
```
運行結果:
```text
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
```
對于常用的屬性比如id，class，可以不用attrs傳遞
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(id='list-1'))
# 由于class是關鍵字需要添加下劃線區分
print(soup.find_all(class_='element'))
```
運行結果:
```text
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
```
#### text {#text}
text 參數可以用來匹配節點的文本，傳入的形式可以是字符串，可以是正則表達式對象
```text
html='''
<div class="panel">
<div class="panel-body">
<a>Hello, this is a link</a>
<a>Hello, this is a link, too</a>
</div>
</div>
'''
import re
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
# 查詢文本包含有link的文本
print(soup.find_all(text=re.compile('link')))
```
運行結果:
```text
['Hello, this is a link', 'Hello, this is a link, too']
```
### find\(\) {#find}
find\(\) 方法返回的是單個元素，即第一個匹配的元素，而 find\_all\(\) 返回的是所有匹配的元素組成的列表
```text
html='''
<div class="panel">
<div class="panel-body">
<a class='element'>Hello, this is a link</a>
<a class='element'>Hello, this is a link, too</a>
</div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.find(name='a'))
print(soup.find(attrs={'class':'element'}))
print(soup.find(class_='element'))
print(type(soup.find(name='a')))
```
返回結果:返回類型為<class 'bs4.element.Tag'>
```text
<a class="element">Hello, this is a link</a>
<a class="element">Hello, this is a link</a>
<a class="element">Hello, this is a link</a>
<class 'bs4.element.Tag'>
```
其他查詢方法
### find\_parents\(\) find\_parent\(\) {#findparents-findparent}
find\_parents\(\) 返回所有祖先節點，find\_parent\(\) 返回直接父節點。
### find\_next\_siblings\(\) find\_next\_sibling\(\) {#findnextsiblings-findnextsibling}
find\_next\_siblings\(\) 返回后面所有兄弟節點，find\_next\_sibling\(\) 返回后面第一個兄弟節點。
### find\_previous\_siblings\(\) find\_previous\_sibling\(\) {#findprevioussiblings-findprevioussibling}
find\_previous\_siblings\(\) 返回前面所有兄弟節點，find\_previous\_sibling\(\) 返回前面第一個兄弟節點。
### find\_all\_next\(\) find\_next\(\) {#findallnext-findnext}
find\_all\_next\(\) 返回節點后所有符合條件的節點, find\_next\(\) 返回第一個符合條件的節點。
### find\_all\_previous\(\) 和 find\_previous\(\) {#findallprevious-和-findprevious}
find\_all\_previous\(\) 返回節點后所有符合條件的節點, find\_previous\(\) 返回第一個符合條件的節點
## 6.CSS選擇器
相關鏈接:[http://www.w3school.com.cn/cssref/css\_selectors.asp](http://www.w3school.com.cn/cssref/css_selectors.asp)。
使用 CSS 選擇器，只需要調用 select\(\) 方法，傳入相應的 CSS 選擇器
實例:
```text
html='''
<div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
print(soup.select('.panel .panel-heading'))
print(soup.select('ul li'))
print(soup.select('#list-2 .element'))
print(soup.select('ul')[0])
```
運行結果:
```text
[<div class="panel-heading">
<h4>Hello</h4>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
```
### 嵌套選擇 {#嵌套選擇}
實例:select\(\) 方法同樣支持嵌套選擇，例如我們先選擇所有 ul 節點，再遍歷每個 ul 節點選擇其 li 節點
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for ul in soup.select('ul'):
print(ul.select('li'))
```
運行結果:
```text
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
```
### 獲取屬性 {#獲取屬性}
獲取屬性還是可以用上面的方法獲取
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for ul in soup.select('ul'):
print(ul['id'])
print(ul.attrs['id'])
```
運行結果:
```text
list-1
list-1
list-2
list-2
```
### 獲取文本 {#獲取文本}
獲取文本可以用string 屬性，還有一種方法那就是 get\_text\(\)，同樣可以獲取文本值。
```text
from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')
for li in soup.select('li'):
print('GET TEXT:',li.get_text())
print('STRING:',li.string)
```
運行結果:
```text
GET TEXT: Foo
STRING: Foo
GET TEXT: Bar
STRING: Bar
GET TEXT: Jay
STRING: Jay
GET TEXT: Foo
STRING: Foo
GET TEXT: Bar
STRING: Bar
```
## 7.細節
* 推薦使用 LXML 解析庫，必要時使用 html.parser
* 節點選擇篩選功能弱但是速度快
* 建議使用 find\(\)、find\_all\(\) 查詢匹配單個結果或者多個結果
* 如果對 CSS 選擇器熟悉的話可以使用 select\(\) 選擇法
如何匹配規則不是熟練，而且想快速獲取，可以如下操作:
右鍵能手敲就手敲，不要偷懶，不然能力提升不上去
- 介紹
- 1.開發環境配置
- 1.1 python3的安裝
- 1.1.1 windows下的安裝
- 1.1.2 Linux下的安裝
- 1.1.3 Mac下的安裝
- 1.2 請求庫的安裝
- 1.2.1 requests的安裝
- 1.2.2 selenium的安裝
- 1.2.3 ChromeDriver的安裝
- 1.2.4 GeckoDriver 的安裝
- 1.2.5 PhantomJS的安裝
- 1.2.6 aiohttp的安裝
- 1.3 解析庫的安裝
- 1.3.1 lxml的安裝
- 1.3.2 Beautiful Soup的安裝
- 1.3.3 pyquery的安裝
- 1.3.4 tesserocr的安裝
- 1.4 數據庫的安裝
- 1.4.1 MySQL的安裝
- 1.4.2 MongoDB的安裝
- 1.4.3 Redis的安裝
- 1.5 存儲庫的安裝
- 1.5.1 PyMySQL的安裝
- 1.5.2 PyMongo的安裝
- 1.5.3 redis-py的安裝
- 1.5.4 RedisDump的安裝
- 1.6 Web庫的安裝
- 1.6.1 Flask的安裝
- 1.6.2 Tornado的安裝
- 1.7 App爬取相關庫的安裝
- 1.7.1 Charles的安裝
- 1.7.2 mitmproxy的安裝
- 1.7.3 Appium的安裝
- 1.8 爬蟲框架的安裝
- 1.8.1 pyspider的安裝
- 1.8.2 Scrapy的安裝
- 1.8.3 Scrapy-Splash的安裝
- 1.8.4 ScrapyRedis的安裝
- 1.9 布署相關庫的安裝
- 1.9.1 Docker的安裝
- 1.9.2 Scrapyd的安裝
- 1.9.3 ScrapydClient的安裝
- 1.9.4 ScrapydAPI的安裝
- 1.9.5 Scrapyrt的安裝
- 1.9.6-Gerapy的安裝
- 2.爬蟲基礎
- 2.1 HTTP 基本原理
- 2.1.1 URI和URL
- 2.1.2 超文本
- 2.1.3 HTTP和HTTPS
- 2.1.4 HTTP請求過程
- 2.1.5 請求
- 2.1.6 響應
- 2.2 網頁基礎
- 2.2.1網頁的組成
- 2.2.2 網頁的結構
- 2.2.3 節點樹及節點間的關系
- 2.2.4 選擇器
- 2.3 爬蟲的基本原理
- 2.3.1 爬蟲概述
- 2.3.2 能抓怎樣的數據
- 2.3.3 javascript渲染的頁面
- 2.4 會話和Cookies
- 2.4.1 靜態網頁和動態網頁
- 2.4.2 無狀態HTTP
- 2.4.3 常見誤區
- 2.5 代理的基本原理
- 2.5.1 基本原理
- 2.5.2 代理的作用
- 2.5.3 爬蟲代理
- 2.5.4 代理分類
- 2.5.5 常見代理設置
- 3.基本庫使用
- 3.1 使用urllib
- 3.1.1 發送請求
- 3.1.2 處理異常
- 3.1.3 解析鏈接
- 3.1.4 分析Robots協議
- 3.2 使用requests
- 3.2.1 基本用法
- 3.2.2 高級用法
- 3.3 正則表達式
- 3.4 抓取貓眼電影排行
- 4.解析庫的使用
- 4.1 使用xpath
- 4.2 使用Beautiful Soup
- 4.3 使用pyquery
- 5.數據存儲
- 5.1 文件存儲
- 5.1.1 TXT 文件存儲
- 5.1.2 JSON文件存儲
- 5.1.3 CSV文件存儲
- 5.2 關系型數據庫存儲
- 5.2.1 MySQL的存儲
- 5.3 非關系數據庫存儲
- 5.3.1 MongoDB存儲
- 5.3.2 Redis存儲
- 6.Ajax數據爬取
- 6.1 什么是Ajax
- 6.2 Ajax分析方法
- 6.3 Ajax結果提取
- 6.4 分析Ajax爬取今日頭條街拍美圖
- 7.動態渲染頁面爬取
- 7.1 Selenium的使用
- 7.2 Splash的使用
- 7.3 Splash負載均衡配置
- 7.4 使用selenium爬取淘寶商品
- 8.驗證碼的識別
- 8.1 圖形驗證碼的識別
- 8.2 極驗滑動驗證碼的識別
- 8.3 點觸驗證碼的識別
- 8.4微博宮格驗證碼的識別
- 9.代理的使用
- 9.1 代理的設置
- 9.2 代理池的維護
- 9.3 付費代理的使用
- 9.4 ADSL撥號代理
- 9.5 使用代理爬取微信公總號文章
- 10.模擬登錄
- 10.1 模擬登陸并爬去GitHub
- 10.2 Cookies池的搭建
- 11.App的爬取
- 11.1 Charles的使用
- 11.2 mitmproxy的使用
- 11.3 mitmdump“得到”App電子書信息
- 11.4 Appium的基本使用
- 11.5 Appnium爬取微信朋友圈
- 11.6 Appium+mitmdump爬取京東商品
- 12.pyspider框架的使用
- 12.1 pyspider框架介紹
- 12.2 pyspider的基本使用
- 12.3 pyspider用法詳解
- 13.Scrapy框架的使用
- 13.1 scrapy框架介紹
- 13.2 入門
- 13.3 selector的用法
- 13.4 spider的用法
- 13.5 Downloader Middleware的用法
- 13.6 Spider Middleware的用法
- 13.7 Item Pipeline的用法
- 13.8 Scrapy對接Selenium
- 13.9 Scrapy對接Splash
- 13.10 Scrapy通用爬蟲
- 13.11 Scrapyrt的使用
- 13.12 Scrapy對接Docker
- 13.13 Scrapy爬取新浪微博
- 14.分布式爬蟲
- 14.1 分布式爬蟲原理
- 14.2 Scrapy-Redis源碼解析
- 14.3 Scrapy分布式實現
- 14.4 Bloom Filter的對接
- 15.分布式爬蟲的部署
- 15.1 Scrapyd分布式部署
- 15.2 Scrapyd-Client的使用
- 15.3 Scrapyd對接Docker
- 15.4 Scrapyd批量部署
- 15.5 Gerapy分布式管理
- 微信公總號文章實戰
- 源碼
- other