httpclient3 · 網絡爬蟲知識匯總

# httpclient3 <div><div style="font-family:Verdana, Arial, Tahoma, sans-serif;font-size:large;font-weight:bold;"><span style="font-size:19px;">第三章 HTTP狀態管理</span></div><div style="font-family:Verdana, Arial, Tahoma, sans-serif;font-size:9.5pt;"><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">原始的HTTP是被設計為無狀態的，面向請求/響應的協議，沒有特殊規定有狀態的，貫穿一些邏輯相關的請求/響應交換的會話。由于HTTP協議變得越來越普及和受歡迎，越來越多的從前沒有打算使用它的系統也開始為應用程序來使用它，比如作為電子商務應用程序的傳輸方式。因此，支持狀態管理就變得非常必要了。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">網景公司，一度成為Web客戶端和服務器軟件開發者的領導方向，在它們基于專有規范的產品中實現了對HTTP狀態管理的支持。之后，網景公司試圖通過發布規范草案來規范這種機制。它們的努力通過RFC標準跟蹤促成了這些規范定義。然而，在很多應用程序中的狀態管理仍然基于網景公司的草案而不兼容官方的規范。很多主要的Web瀏覽器開發者覺得有必要保留那些極大促進標準片段應用程序的兼容性。</p><h3>3.1 HTTP cookies</h3><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">Cookie是HTTP代理和目標服務器可以交流保持會話的狀態信息的令牌或短包。網景公司的工程師用它來指“魔法小甜餅”和粘住的名字。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient 使用Cookie接口來代表抽象的cookie令牌。在它的簡單形式中HTTP的cookie幾乎是名/值對。通常一個HTTP的cookie也包含一些屬性，比如版本號，合法的域名，指定cookie應用所在的源服務器URL子集的路徑，cookie的最長有效時間。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">SetCookie接口代表由源服務器發送給HTTP代理的響應中的頭部信息Set-Cookie來維持一個對話狀態。SetCookie2接口和指定的Set-Cookie2方法擴展了SetCookie。</p><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">SetCookie 接口和額外的如獲取原始cookie屬性的能力，就像它們由源服務器指定的客戶端特定功能擴展了Cookie接口。這對生成Cookie頭部很重要，因為一些cookie規范需要。Cookie頭部應該包含在Set-Cookie或Set-Cookie2頭部中指定的特定屬性。</p><h4>3.1.1 Cookie版本</h4><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">Cookie兼容網景公司的草案標準，但是版本0被認為是不符合官方規范的。符合標準的cookie的期望版本是1。HttpClient可以處理基于不同版本的cookie。</div><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">這里有一個重新創建網景公司草案cookie示例：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">BasicClientCookie netscapeCookie = new BasicClientCookie("name", "value");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">netscapeCookie.setVersion(0);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">netscapeCookie.setDomain(".mycompany.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">netscapeCookie.setPath("/");</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">這是一個重新創建標準cookie的示例。要注意符合標準的cookie必須保留由源服務器發送的所有屬性：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">BasicClientCookie stdCookie = new BasicClientCookie("name", "value");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setVersion(1);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setDomain(".mycompany.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setPath("/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setSecure(true);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 精確設置由服務器發送的屬性</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setAttribute(ClientCookie.VERSION_ATTR, "1");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setAttribute(ClientCookie.DOMAIN_ATTR, ".mycompany.com");</div></blockquote><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">這是一個重新創建Set-Cookie2兼容cookie的實例。要注意符合標準的cookie必須保留由源服務器發送的所有屬性：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">BasicClientCookie2 stdCookie = new BasicClientCookie2("name", "value");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setVersion(1);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setDomain(".mycompany.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setPorts(new int[] {80,8080});</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setPath("/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setSecure(true);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 精確設置由服務器發送的屬性</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setAttribute(ClientCookie.VERSION_ATTR, "1");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setAttribute(ClientCookie.DOMAIN_ATTR, ".mycompany.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">stdCookie.setAttribute(ClientCookie.PORT_ATTR, "80,8080");</div></blockquote><h3>3.2 Cookie規范</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieSpec接口代表了cookie管理的規范。Cookie管理規范希望如下幾點：</div><ul><li><span style="color:#888888;">解析的Set-Cookie規則還有可選的Set-Cookie2頭部信息。</span></li><li><span style="color:#888888;">驗證解析cookie的規則。</span></li><li><span style="color:#888888;">格式化給定主機的Cookie頭部信息，原始端口和路徑。</span></li></ul><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient附帶了一些CookieSpec的實現：</p><ul><li><span style="color:#888888;">網景公司草案：這個規范符合由網景通訊發布的原始草案規范。應當避免，除非有絕對的必要去兼容遺留代碼。</span></li><li><span style="color:#888888;">RFC 2109：官方HTTP狀態管理規范并取代的老版本，被RFC 2965取代。</span></li><li><span style="color:#888888;">RFC 2965：官方HTTP狀態管理規范。</span></li><li><span style="color:#888888;">瀏覽器兼容性：這個實現努力去密切模仿（mis）通用Web瀏覽器應用程序的實現。比如微軟的Internet Explorer和Mozilla的FireFox瀏覽器。</span></li><li><span style="color:#888888;">最佳匹配：’Meta’（元）cookie規范采用了一些基于又HTTP響應發送的cookie格式的cookie策略。它基本上聚合了以上所有的實現到以一個類中。</span></li></ul><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">強烈建議使用Best Match策略，讓HttpClient在運行時基于執行上下文采用一些合適的兼容等級。</div><h3>3.3 HTTP cookie和狀態管理參數</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">這些是用于定制HTTP狀態管理和獨立的cookie規范行為的參數。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><ul><li><span style="color:#888888;">'http.protocol.cookie- datepatterns'：定義了用于解析非標準的expires屬性的合法日期格式。只是對兼容不符合規定的，仍然使用網景公司草案定義的 expires而不使用標準的max-age屬性服務器需要。這個參數期望得到一個java.util.Collection類型的值。集合元素必須是 java.lang.String類型，來兼容java.text.SimpleDateFormat的語法。如果這個參數沒有被設置，那么默認的選擇就是CookieSpec實現規范的值。要注意這個參數的應用。</span></li><li><span style="color:#888888;">'http.protocol.single- cookie-header'：定義了是否cookie應該強制到一個獨立的Cookie請求頭部信息中。否則，每個cookie就被當作分離的 Cookie頭部信息來格式化。這個參數期望得到一個java.lang.Boolean類型的值。如果這個參數沒有被設置，那么默認的選擇就是 CookieSpec實現規范的值。要注意這個參數僅僅嚴格應用于cookie規范（RFC 2109和RFC 2965）。瀏覽器兼容性和網景公司草案策略將會放置所有的cookie到一個請求頭部信息中。</span></li><li><span style="color:#888888;">'http.protocol.cookie-policy'：定義了用于HTTP狀態管理的cookie規范的名字。這個參數期望得到一個java.lang.String類型的值。如果這個參數沒有被設置，那么合法的日期格式就是CookieSpec實現規范的值。</span></li></ul><h3>3.4 Cookie規范注冊表</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient使用CookieSpecRegistry類維護一個可用的cookie規范注冊表。下面的規范對于每個默認都是注冊過的：</div><ul><li><span style="color:#888888;">兼容性：瀏覽器兼容性（寬松策略）。</span></li><li><span style="color:#888888;">網景：網景公司草案。</span></li><li><span style="color:#888888;">rfc2109：RFC 2109（過時的嚴格策略）。</span></li><li><span style="color:#888888;">rfc2965：RFC 2965（嚴格策略的標準符合）。</span></li><li><span style="color:#888888;">best-match：最佳匹配meta（元）策略。</span></li></ul><h3>3.5 選擇cookie策略</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">Cookie策略可以在HTTP客戶端被設置，如果需要，在HTTP請求級重寫。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 對每個默認的強制嚴格cookie策略</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.getParams().setParameter(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientPNames.COOKIE_POLICY, CookiePolicy.RFC_2965);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpGet httpget = new HttpGet("http://www.broken-server.com/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 對這個請求覆蓋默認策略</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpget.getParams().setParameter(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientPNames.COOKIE_POLICY, CookiePolicy.BROWSER_COMPATIBILITY);</div></blockquote><h3>3.6 定制cookie策略</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">為了實現定制cookie策略，我們應該創建CookieSpec接口的定制實現類，創建一個CookieSpecFactory實現來創建和初始化定制實現的實例并和HttpClient注冊這個工廠。一旦定制實現被注冊了，它可以和標準的cookie實現有相同的活性。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieSpecFactory csf = new CookieSpecFactory() {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">public CookieSpec newInstance(HttpParams params) {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">return new BrowserCompatSpec() {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">@Override</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">public void validate(Cookie cookie, CookieOrigin origin)</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">throws MalformedCookieException {</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 這相當簡單</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">};</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">}</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">};</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">DefaultHttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.getCookieSpecs().register("easy", csf);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.getParams().setParameter(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientPNames.COOKIE_POLICY, "easy");</div></blockquote><h3>3.7 Cookie持久化</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient 可以和任意物理表示的實現了CookieStore接口的持久化cookie存儲一起使用。默認的CookieStore實現稱為 BasicClientCookie，這是憑借java.util.ArrayList的一個簡單實現。在BasicClientCookie對象中存儲的cookie當容器對象被垃圾回收機制回收時會丟失。如果需要，用戶可以提供更復雜的實現。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">DefaultHttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 創建一個本地的cookie store實例</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieStore cookieStore = new MyCookieStore();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 如果需要填充cookie</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">BasicClientCookie cookie = new BasicClientCookie("name", "value");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">cookie.setVersion(0);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">cookie.setDomain(".mycompany.com");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">cookie.setPath("/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">cookieStore.addCookie(cookie);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 設置存儲</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">httpclient.setCookieStore(cookieStore);</div></blockquote><h3>3.8 HTTP狀態管理和執行上下文</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">在HTTP請求執行的過程中，HttpClient添加了下列和狀態管理相關的對象到執行上下文中：</div><ul><li><span style="color:#888888;">'http.cookiespec-registry'：CookieSpecRegistry實例代表了實際的cookie規范注冊表。這個屬性的值設置在本地內容中，優先于默認的。</span></li><li><span style="color:#888888;">'http.cookie-spec'：CookieSpec實例代表真實的cookie規范。</span></li><li><span style="color:#888888;">'http.cookie-origin'：CookieOrigin實例代表了真實的源服務器的詳細信息。</span></li><li><span style="color:#888888;">'http.cookie-store'：CookieStore實例代表了真實的cookie存儲。設置在本地內容中的這個屬性的值優先于默認的。</span></li></ul><p style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">本地的HttpContext對象可以被用來定制HTTP狀態管理內容，先于請求執行或在請求執行之后檢查它的狀態：</p><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpContext localContext = new BasicHttpContext();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpGet httpget = new HttpGet("http://localhost:8080/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpResponse response = httpclient.execute(httpget, localContext);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieOrigin cookieOrigin = (CookieOrigin) localContext.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientContext.COOKIE_ORIGIN);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Cookie origin: " + cookieOrigin);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieSpec cookieSpec = (CookieSpec) localContext.getAttribute(</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">ClientContext.COOKIE_SPEC);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">System.out.println("Cookie spec used: " + cookieSpec);</div></blockquote><h3>3.9 每個用戶/線程的狀態管理</h3><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">我們可以使用獨立的本地執行上下文來實現對每個用戶（或每個線程）狀態的管理。定義在本地內容中的cookie規范注冊表和cookie存儲將會優先于設置在HTTP客戶端級別中默認的那些。</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;"><blockquote style="background-color:#F9F9FF;"><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpClient httpclient = new DefaultHttpClient();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 創建cookie store的本地實例</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">CookieStore cookieStore = new BasicCookieStore();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 創建本地的HTTP內容</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpContext localContext = new BasicHttpContext();</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 綁定定制的cookie store到本地內容中</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpGet httpget = new HttpGet("http://www.google.com/");</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">// 作為參數傳遞本地內容</div><div style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;">HttpResponse response = httpclient.execute(httpget, localContext)</div></blockquote></div></div></div></div></div></div></div> > 資源參考 http://www.cnblogs.com/loveyakamoz/archive/2011/07/21/2112804.html