7.1 Selenium的使用 · python3爬蟲筆記

### 1.說明 selenium是一個自動化測試工具，可以利用它驅動瀏覽器執行特定的動作中文文檔:[http://selenium-python-zh.readthedocs.io/en/latest/](http://selenium-python-zh.readthedocs.io/en/latest/) 英文文檔:[https://seleniumhq.github.io/selenium/docs/api/py/api.htm](https://seleniumhq.github.io/selenium/docs/api/py/api.html)l 節點操作:[http://selenium-python.readthedocs.io/api.html\#module-selenium.webdriver.remote.webelement](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement) 動作鏈:[http://selenium-python.readthedocs.io/api.html\#module-selenium.webdriver.common.action\_chains](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.common.action_chains) 等待條件參數及用法介紹:[http://selenium-python.readthedocs.io/api.html\#module-selenium.webdriver.support.expected\_conditions](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.support.expected_conditions) 異常類:[http://selenium-python.readthedocs.io/api.html\#module-selenium.common.exceptions](http://selenium-python.readthedocs.io/api.html#module-selenium.common.exceptions) ### 2.準備 [安裝selenium及其驅動](/1kai-fa-huan-jing-pei-zhi/12-qing-qiu-ku-de-an-zhuang/122-seleniumde-an-zhuang.md) ### 3.基本使用 ``` from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait browser = webdriver.Chrome() try: browser.get('https://www.baidu.com') input = browser.find_element_by_id('kw') input.send_keys("python") input.send_keys(Keys.ENTER) wait = WebDriverWait(browser,10) wait.until(EC.presence_of_element_located((By.ID,"content_left"))) print(browser.current_url) print(browser.get_cookies()) print(browser.page_source) except: browser.close() ``` ### 3.聲明瀏覽器對象 ``` from selenium import webdriver browser = webdriver.Chrome() browser = webdriver.Firefox() browser = webdriver.Edge() browser = webdriver.PhantomJS() browser = webdriver.Safari() ``` ### 4.訪問頁面調用get\(\)方法請求第一個網頁，參數傳入鏈接url，通過page\_source屬性獲取源碼 ``` browser.get('https://www.baidu.com') print(browser.page_source) ``` ### 5.查找節點 #### 單個結點獲取淘寶搜索框的節點 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.taobao.com") # 根據id獲取 input_first = browser.find_element_by_id("q") # 根據name獲取 input_second = browser.find_element_by_name("q") # css選擇器 input_third = browser.find_element_by_css_selector("#q") # xpath input_four = browser.find_element_by_xpath("//*[@id='q']") print(input_first,input_second,input_third,input_four,sep='\n') browser.close() ``` ``` find_element_by_id <==> find_element(ByID,id) find_element_by_name find_element_by_xpath find_element_by_link_text find_element_by_partial_link_text find_element_by_tag_name find_element_by_class_name find_element_by_css_selector ``` 多個節點 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.taobao.com") lis = browser.find_elements_by_css_selector(".service-bd li") print(lis) browser.close() ``` ``` find_elements_by_id find_elements_by_name find_elements_by_xpath find_elements_by_link_text find_elements_by_partial_link_text find_elements_by_tag_name find_elements_by_class_name find_elements_by_css_selector <==> find.elements(By.CSS_SELECTOR,'.service-bd li') ``` ### 6.節點交互 send\_key\(\)：輸入 clear\(\)：清空 click\(\)：點擊 ``` from selenium import webdriver import time browser = webdriver.Chrome() browser.get("https://www.taobao.com/") # 選中輸入框 input = browser.find_element_by_css_selector("#q") input.clear() # 輸入 input.send_keys("蘋果") # 等待時間 time.sleep(1) # 清空 input.clear() # 輸入 input.send_keys("香蕉") # 選中按鈕 button = browser.find_element_by_css_selector("#J_TSearchForm > div.search-button > button") # 點擊 button.click() ``` ### 7.動作鏈 ``` from selenium import webdriver from selenium.webdriver import ActionChains browser = webdriver.Chrome() url = "http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable" browser.get(url) # 根據id切換iframe # browser.switch_to_frame("iframeResult") # 最新的切換方法 browser.switch_to.frame("iframeResult") source = browser.find_element_by_css_selector("#draggable") target = browser.find_element_by_css_selector("#droppable") actions = ActionChains(browser) # 調用拖拽方法 actions.drag_and_drop(source,target) # 執行動作 actions.perform() ``` ### 8.執行javascript 模擬執行javascript，使用execute\_script\(\)方法實現 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.zhihu.com/explore") # 將進度條下拉到最底部 browser.execute_script('window.scrollTo(0, document.body.scrollHeight)') # 彈出提示框 browser.execute_script('alert("To Bottom")') ``` ### 9.獲取節點信息 #### 獲取屬性使用get\_attribute\(\)方法來獲取節點的屬性 ``` from selenium import webdriver browser = webdriver.Chrome() url = "https://www.zhihu.com/explore" browser.get(url) logo = browser.find_element_by_id("zh-top-link-logo") print(logo) print(logo.get_attribute("class")) ``` 運行結果: ``` <selenium.webdriver.remote.webelement.WebElement (session="036f1edccab08bbc5ca2b8f02accbb04", element="0.40649588367097245-1")> zu-top-link-logo ``` #### 獲取文本值通過text屬性獲取文本信息 ``` from selenium import webdriver browser = webdriver.Chrome() url = "https://www.zhihu.com/explore" browser.get(url) input = browser.find_element_by_id("zu-top-add-question") print(input.text) ``` 運行結果: ``` 提問 ``` #### 獲取ID、位置、標簽名、大小 id屬性:獲取節點id location:獲取節點再頁面中的相對位置 tag\_name:獲取標簽名稱 size:獲取節點的大小 ``` from selenium import webdriver browser = webdriver.Chrome() url = "https://www.zhihu.com/explore" browser.get(url) input = browser.find_element_by_id("zu-top-add-question") print(input.id) print(input.location) print(input.tag_name) print(input.size) ``` 運行結果: ``` 0.4994837841126192-1 {'x': 683, 'y': 7} button {'height': 32, 'width': 66} ``` ### 10.切換Frame 使用switch\_to.frame\(\)切換frame 示例:QQ郵箱登錄 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://mail.qq.com/") # 等待三秒 browser.implicitly_wait(3) # 因為頁面有很多iframe的內置框架，需要先逐級定位到登錄元素所在的iframe才行 browser.switch_to.frame("login_frame") name = browser.find_element_by_name("u") name.send_keys("xxxx@qq.com") password = browser.find_element_by_name("p") password.send_keys("xxxx") button = browser.find_element_by_id("login_button") button.click() ``` ### 11.延時等待 #### 隱式等待 implicitly\_wait\(\):等待幾秒 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://mail.qq.com/") browser.implicitly_wait(3) browser.switch_to.frame("login_frame") name = browser.find_element_by_name("u") name.send_keys("xxxx@qq.com") password = browser.find_element_by_name("p") password.send_keys("xxxx") button = browser.find_element_by_id("login_button") button.click() ``` #### 顯示等待 ``` from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions from selenium.webdriver.common.by import By browser = webdriver.Chrome() browser.get("https://mail.qq.com/") # 創建WebDriverWait實例化對象，并指定最長等待時間 wait = WebDriverWait(browser,3) browser.switch_to.frame("login_frame") # presence_of_element_located:查詢節點，參數為結點的定位元組 name = wait.until(expected_conditions.presence_of_element_located((By.ID,'u'))) name.send_keys("xxx@qq.com") password = browser.find_element_by_name("p") password.send_keys("xxx") # element_to_be_clickable:可執行點擊 button = wait.until(expected_conditions.element_to_be_clickable((By.ID,'login_button'))) button.click() ``` #### 所有的加載條件 | 等待條件 | 含義 | | :--- | :--- | | title\_is | 標題是某內容 | | title\_contains | 標題包含某內容 | | presence\_of\_element\_located | 節點加載出，傳入定位元組，如\(By.ID, 'p'\) | | visibility\_of\_element\_located | 節點可見，傳入定位元組 | | visibility\_of | 可見，傳入節點對象 | | presence\_of\_all\_elements\_located | 所有節點加載出 | | text\_to\_be\_present\_in\_element | 某個節點文本包含某文字 | | text\_to\_be\_present\_in\_element\_value | 某個節點值包含某文字 | | frame\_to\_be\_available\_and\_switch\_to\_it frame | 加載并切換 | | invisibility\_of\_element\_located | 節點不可見 | | element\_to\_be\_clickable | 節點可點擊 | | staleness\_of | 判斷一個節點是否仍在DOM，可判斷頁面是否已經刷新 | | element\_to\_be\_selected | 節點可選擇，傳節點對象 | | element\_located\_to\_be\_selected | 節點可選擇，傳入定位元組 | | element\_selection\_state\_to\_be | 傳入節點對象以及狀態，相等返回True，否則返回False | | element\_located\_selection\_state\_to\_be | 傳入定位元組以及狀態，相等返回True，否則返回False | | alert\_is\_present | 是否出現Alert | ### 12.前進后退 back\(\):后退 forward\(\):前進 ``` import time from selenium import webdriver browser = webdriver.Chrome() browser.get('https://www.baidu.com/') browser.get('https://www.taobao.com/') browser.get('https://www.python.org/') browser.back() time.sleep(1) browser.forward() browser.close() ``` ### 13.Cookies 對cookies進行操作 ``` from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.zhihu.com/explore") print(browser.get_cookies()) # 添加cookies browser.add_cookie({"name":"angle","value":"value"}) # 獲取cookies print(browser.get_cookies()) # 刪除所有cookies browser.delete_all_cookies() # 獲取cookies print(browser.get_cookies()) ``` 運行結果: ``` [{'domain': '.zhihu.com', 'httpOnly': False, 'name': 'l_n_c', 'path': '/', 'secure': False, 'value': '1'}, {'domain': 'www.zhihu.com', 'expiry': 1535626235.952171, 'httpOnly': False, 'name': 'tgw_l7_route', 'path': '/', 'secure': False, 'value': '56f3b730f2eb8b75242a8095a22206f8'}, {'domain': '.zhihu.com', 'expiry': 1535627141, 'httpOnly': False, 'name': '__utmb', 'path': '/', 'secure': False, 'value': '51854390.0.10.1535625342'}, {'domain': '.zhihu.com', 'expiry': 1630233335.95244, 'httpOnly': False, 'name': 'q_c1', 'path': '/', 'secure': False, 'value': 'df753284d0ae4a2cb8bda8e41bc190e8|1535625336000|1535625336000'}, {'domain': 'www.zhihu.com', 'httpOnly': False, 'name': '_xsrf', 'path': '/', 'secure': False, 'value': '4c00cb0973af088c42d4a58468817612'}, {'domain': '.zhihu.com', 'expiry': 1538217335.95261, 'httpOnly': False, 'name': 'r_cap_id', 'path': '/', 'secure': False, 'value': '"OGEwOWQzNGYxYTZiNGU0ZDhlMTJmNGFhZmZhMjJjZTQ=|1535625336|37bd3be47d98ccd1123d7e35bf76cb1312d7143b"'}, {'domain': '.zhihu.com', 'expiry': 1538217335.952672, 'httpOnly': False, 'name': 'cap_id', 'path': '/', 'secure': False, 'value': '"MzJmZWJlNDc1YjY0NGQ0OGEzMGI4ZTUwMGE4OThhZmU=|1535625336|a4628b2bd1054ddfed0bb5f0b02cf7fbca1e5216"'}, {'domain': '.zhihu.com', 'expiry': 1538217335.952705, 'httpOnly': False, 'name': 'l_cap_id', 'path': '/', 'secure': False, 'value': '"ZGU4YTdlNjJhMWRiNDkyZGE3Y2I3OTYzYzkxN2IzMzg=|1535625336|dc1242668bfb5a0d13df384da1248835b6a1d7ba"'}, {'domain': '.zhihu.com', 'httpOnly': False, 'name': 'n_c', 'path': '/', 'secure': False, 'value': '1'}, {'domain': '.zhihu.com', 'expiry': 1630233340.416983, 'httpOnly': False, 'name': 'd_c0', 'path': '/', 'secure': False, 'value': '"AAAmfnWtIg6PTicvdBW52HutZ46oS_fCo1Y=|1535625341"'}, {'domain': '.zhihu.com', 'expiry': 1598697340, 'httpOnly': False, 'name': '_zap', 'path': '/', 'secure': False, 'value': '55900a43-f801-4abe-b643-1f67dd005a6e'}, {'domain': '.zhihu.com', 'expiry': 1598697341, 'httpOnly': False, 'name': '__utma', 'path': '/', 'secure': False, 'value': '51854390.1748932163.1535625342.1535625342.1535625342.1'}, {'domain': '.zhihu.com', 'httpOnly': False, 'name': '__utmc', 'path': '/', 'secure': False, 'value': '51854390'}, {'domain': '.zhihu.com', 'expiry': 1551393341, 'httpOnly': False, 'name': '__utmz', 'path': '/', 'secure': False, 'value': '51854390.1535625342.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'}, {'domain': '.zhihu.com', 'expiry': 1598697341, 'httpOnly': False, 'name': '__utmv', 'path': '/', 'secure': False, 'value': '51854390.000--|3=entry_date=20180830=1'}] [{'domain': '.zhihu.com', 'httpOnly': False, 'name': 'l_n_c', 'path': '/', 'secure': False, 'value': '1'}, {'domain': 'www.zhihu.com', 'expiry': 1535626235.952171, 'httpOnly': False, 'name': 'tgw_l7_route', 'path': '/', 'secure': False, 'value': '56f3b730f2eb8b75242a8095a22206f8'}, {'domain': '.zhihu.com', 'expiry': 1535627141, 'httpOnly': False, 'name': '__utmb', 'path': '/', 'secure': False, 'value': '51854390.0.10.1535625342'}, {'domain': '.zhihu.com', 'expiry': 1630233335.95244, 'httpOnly': False, 'name': 'q_c1', 'path': '/', 'secure': False, 'value': 'df753284d0ae4a2cb8bda8e41bc190e8|1535625336000|1535625336000'}, {'domain': 'www.zhihu.com', 'httpOnly': False, 'name': '_xsrf', 'path': '/', 'secure': False, 'value': '4c00cb0973af088c42d4a58468817612'}, {'domain': '.zhihu.com', 'expiry': 1538217335.95261, 'httpOnly': False, 'name': 'r_cap_id', 'path': '/', 'secure': False, 'value': '"OGEwOWQzNGYxYTZiNGU0ZDhlMTJmNGFhZmZhMjJjZTQ=|1535625336|37bd3be47d98ccd1123d7e35bf76cb1312d7143b"'}, {'domain': '.zhihu.com', 'expiry': 1538217335.952672, 'httpOnly': False, 'name': 'cap_id', 'path': '/', 'secure': False, 'value': '"MzJmZWJlNDc1YjY0NGQ0OGEzMGI4ZTUwMGE4OThhZmU=|1535625336|a4628b2bd1054ddfed0bb5f0b02cf7fbca1e5216"'}, {'domain': '.zhihu.com', 'expiry': 1538217335.952705, 'httpOnly': False, 'name': 'l_cap_id', 'path': '/', 'secure': False, 'value': '"ZGU4YTdlNjJhMWRiNDkyZGE3Y2I3OTYzYzkxN2IzMzg=|1535625336|dc1242668bfb5a0d13df384da1248835b6a1d7ba"'}, {'domain': '.zhihu.com', 'httpOnly': False, 'name': 'n_c', 'path': '/', 'secure': False, 'value': '1'}, {'domain': '.zhihu.com', 'expiry': 1630233340.416983, 'httpOnly': False, 'name': 'd_c0', 'path': '/', 'secure': False, 'value': '"AAAmfnWtIg6PTicvdBW52HutZ46oS_fCo1Y=|1535625341"'}, {'domain': '.zhihu.com', 'expiry': 1598697340, 'httpOnly': False, 'name': '_zap', 'path': '/', 'secure': False, 'value': '55900a43-f801-4abe-b643-1f67dd005a6e'}, {'domain': '.zhihu.com', 'expiry': 1598697341, 'httpOnly': False, 'name': '__utma', 'path': '/', 'secure': False, 'value': '51854390.1748932163.1535625342.1535625342.1535625342.1'}, {'domain': '.zhihu.com', 'httpOnly': False, 'name': '__utmc', 'path': '/', 'secure': False, 'value': '51854390'}, {'domain': '.zhihu.com', 'expiry': 1551393341, 'httpOnly': False, 'name': '__utmz', 'path': '/', 'secure': False, 'value': '51854390.1535625342.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'}, {'domain': '.zhihu.com', 'expiry': 1598697341, 'httpOnly': False, 'name': '__utmv', 'path': '/', 'secure': False, 'value': '51854390.000--|3=entry_date=20180830=1'}, {'domain': 'www.zhihu.com', 'expiry': 2166345341, 'httpOnly': False, 'name': 'angle', 'path': '/', 'secure': True, 'value': 'value'}] [] ``` ### 14.選項卡管理切換窗口:switch\_to.window\(browser.window.handles\[x\]\) ``` # from selenium import webdriver # # browser = webdriver.Chrome() # browser.get("https://www.zhihu.com/explore") # print(browser.get_cookies()) # # 添加cookies # browser.add_cookie({"name":"angle","value":"value"}) # # 獲取cookies # print(browser.get_cookies()) # # 刪除所有cookies # browser.delete_all_cookies() # # 獲取cookies # print(browser.get_cookies()) import time from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.baidu.com") browser.execute_script("window.open()") print(browser.window_handles) # 切換選項卡 browser.switch_to.window(browser.window_handles[1]) browser.get("https://www.taobao.com") time.sleep(1) browser.switch_to.window(browser.window_handles[0]) browser.get("https://python.org")# from selenium import webdriver # # browser = webdriver.Chrome() # browser.get("https://www.zhihu.com/explore") # print(browser.get_cookies()) # # 添加cookies # browser.add_cookie({"name":"angle","value":"value"}) # # 獲取cookies # print(browser.get_cookies()) # # 刪除所有cookies # browser.delete_all_cookies() # # 獲取cookies # print(browser.get_cookies()) import time from selenium import webdriver browser = webdriver.Chrome() browser.get("https://www.baidu.com") browser.execute_script("window.open()") print(browser.window_handles) # 切換選項卡 browser.switch_to.window(browser.window_handles[1]) browser.get("https://www.taobao.com") time.sleep(1) browser.switch_to.window(browser.window_handles[0]) browser.get("https://python.org") ``` 運行結果: ``` ['CDwindow-7A6C2399B267FCE1C8E4A69C49D515B3', 'CDwindow-A751A13C930C1383E6E228CD4C44CEA1'] ``` ### 15.異常處理 ``` from selenium import webdriver from selenium.common.exceptions import TimeoutException, NoSuchElementException browser = webdriver.Chrome() try: browser.get('https://www.baidu.com') except TimeoutException: print('Time Out') try: browser.find_element_by_id('hello') except NoSuchElementException: print('No Element') finally: browser.close() ``` 運行結果: ``` No Element ```