elasticsearch中的各種查詢 · 碼山有道：Java技術生態

[toc] # ElaticSearch ## 3.ElasticSearch練習 - 索引 ： sms-logs-index - 類型：sms-logs-type | 字段名稱 | 備注 | | ---------- | ---------------------------------------------- | | createDate | 創建時間String | | sendDate | 發送時間 date | | longCode | 發送長號碼如 16092389287811 string | | Mobile | 如 13000000000 | | corpName | 發送公司名稱，需要分詞檢索 | | smsContent | 下發短信內容，需要分詞檢索 | | State | 短信下發狀態 0 成功 1 失敗 integer | | Operatorid | 運營商編號1移動2聯通3電信 integer | | Province | 省份 | | ipAddr | 下發服務器IP地址 | | replyTotal | 短信狀態報告返回時長 integer | | Fee | 扣費 integer | | | | - 創建實例代碼 ~~~java //先定義索引名和類型名 String index = "sms_logs_index"; String type = "sms_logs_type"; ~~~ ```java public void create_index() throws IOException { Settings.Builder settings = Settings.builder() .put("number_of_shards", 3) .put("number_of_replicas", 1); XContentBuilder mappings = JsonXContent.contentBuilder() .startObject() .startObject("properties") .startObject("createDate") .field("type", "text") .endObject() .startObject("sendDate") .field("type", "date") .field("format", "yyyy-MM-dd") .endObject() .startObject("longCode") .field("type", "text") .endObject() .startObject("mobile") .field("type", "text") .endObject() .startObject("corpName") .field("type", "text") .field("analyzer", "ik_max_word") .endObject() .startObject("smsContent") .field("type", "text") .field("analyzer", "ik_max_word") .endObject() .startObject("state") .field("type", "integer") .endObject() .startObject("operatorid") .field("type", "integer") .endObject() .startObject("province") .field("type", "text") .endObject() .startObject("ipAddr") .field("type", "text") .endObject() .startObject("replyTotal") .field("type", "integer") .endObject() .startObject("fee") .field("type", "integer") .endObject() .endObject() .endObject(); CreateIndexRequest request = new CreateIndexRequest(index) .settings(settings) .mapping(type,mappings); RestHighLevelClient client = ESClient.getClient(); CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT); System.out.println(response.toString()); } ``` - 數據導入部分 ```json PUT /sms_logs_index/sms_logs_type/1 { "corpName": "途虎養車", "createDate": "2020-1-22", "fee": 3, "ipAddr": "10.123.98.0", "longCode": 106900000009, "mobile": "1738989222222", "operatorid": 1, "province": "河北", "relyTotal": 10, "sendDate": "2020-2-22", "smsContext": "【途虎養車】親愛的燈先生，您的愛車已經購買", "state": 0 } ``` ## 4. ES的各種查詢 ### 4.1 term&terms查詢 #### 4.1.1 term查詢 - ? term的查詢是代表完全匹配，搜索之前不會對你的關鍵字進行分詞 ```json #term匹配查詢 POST /sms_logs_index/sms_logs_type/_search { "from": 0, #limit from,size "size": 5, "query": { "term": { "province": { "value": "河北" } } } } ##不會對term中所匹配的值進行分詞查詢 ``` ```java // java代碼實現方式 @Test public void testQuery() throws IOException { // 1 創建Request對象 SearchRequest request = new SearchRequest(index); request.types(type); // 2 指定查詢條件 SearchSourceBuilder builder = new SearchSourceBuilder(); builder.from(0); builder.size(5); builder.query(QueryBuilders.termQuery("province", "河北")); request.source(builder); // 3 執行查詢 RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); // 4 獲取到_source中的數據 for (SearchHit hit : response.getHits().getHits()) { Map<String, Object> result = hit.getSourceAsMap(); System.out.println(result); } } ``` - terms是針對一個字段包含多個值得運用 - terms: where province = 河北 or province = ? or province = ? ```json #terms 匹配查詢 POST /sms_logs_index/sms_logs_type/_search { "from": 0, "size": 5, "query": { "terms": { "province": [ "河北", "河南" ] } } } ``` ```java // java代碼 terms 查詢 @Test public void test_terms() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.termsQuery("province","河北","河南")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse resp = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : resp.getHits().getHits()){ System.out.println(hit); } } ``` ### 4.2 match查詢 match查詢屬于高層查詢，它會根據你查詢字段類型不一樣，采用不同的查詢方式 match查詢，實際底層就是多個term查詢，將多個term查詢的結果進行了封裝 - 查詢的如果是日期或者是數值的話，它會根據你的字符串查詢內容轉換為日期或者是數值對等 - 如果查詢的內容是一個不可被分的內容（keyword），match查詢不會對你的查詢的關鍵字進行分詞 - 如果查詢的內容是一個可被分的內容（text）,match則會根據指定的查詢內容按照一定的分詞規則去分詞進行查詢 #### 4.2.1 match_all查詢 查詢全部內容，不指定任何查詢條件 ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "match_all": {} } } ~~~ ~~~java @Test public void test_match_all() throws IOException { // 創建Request ,放入索引和類型 SearchRequest request = new SearchRequest(index); request.types(type); builder.size(20); //es默認查詢結果只展示10條，這里可以指定展示的條數 //指定查詢條件 SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.matchAllQuery()); request.source(builder); // 執行查詢 RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); // 獲取查詢結果，遍歷顯示 for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit); } } ~~~ #### 4.2.2 match查詢根據某個Field ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "match": { "smsContent": "打車" } } } ~~~ ~~~java @Test public void test_match_field() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.matchQuery("smsContext","打車")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit); } } ~~~ #### 4.2.3 布爾match查詢 基于一個Filed匹配的內容，采用and或者or的方式進行連接 ~~~json # 布爾match查詢 POST /sms_logs_index/sms_logs_type/_search { "query": { "match": { "smsContext": { "query": "打車女士", "operator": "and" #or } } } } ~~~ ~~~java @Test public void test_match_boolean() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.matchQuery("smsContext","打車女士").operator(Operator.AND)); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit); } ~~~ #### 4.2.4 multi_match查詢 match針對一個field做檢索，multi_match針對多個field進行檢索，多個key對應一個text ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "multi_match": { "query": "河北", #指定text "fields": ["province","smsContext"] #指定field } } } ~~~ ~~~java // java 實現 @Test public void test_multi_match() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); // 查詢的文本內容字段1 字段2 字段3 。。。。。 builder.query(QueryBuilders.multiMatchQuery("河北", "province", "smsContext")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()) { System.out.println(hit); } } ~~~ ### 4.3 ES 的其他查詢 #### 4.3.1 ID 查詢 ~~~JSON # id查詢 GET /sms_logs_index/sms_logs_type/1 GET /索引名/type類型/id ~~~ ~~~java public void test_multi_match() throws IOException { GetRequest request = new GetRequest(index,type,"1"); RestHighLevelClient client = ESClient.getClient(); GetResponse resp = client.get(request, RequestOptions.DEFAULT); System.out.println(resp.getSourceAsMap()); } ~~~ #### 4.3.2 ids查詢 根據多個id進行查詢，類似MySql中的where Id in (id1,id2,id3….) ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "ids": { "values": [1,2,3] #id值 } } } ~~~ ~~~java //java代碼 @Test public void test_query_ids() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.idsQuery().addIds("1","2","3")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } ~~~ #### 4.3.3 prefix查詢 前綴查詢，可以通過一個關鍵字去指定一個Field的前綴，從而查詢到指定的文檔 ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "prefix": { "smsContext": { "value": "河" } } } } #與 match查詢的不同在于，prefix類似mysql中的模糊查詢。而match的查詢類似于嚴格匹配查詢 # 針對不可分割詞 ~~~ ~~~java @Test public void test_query_prefix() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.prefixQuery("smsContext","河")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } ~~~ #### 4.3.4 fuzzy查詢 fuzzy查詢：模糊查詢，我們可以輸入一個字符的大概，ES就可以根據輸入的內容大概去匹配一下結果，eg.你可以存在一些錯別字 ~~~json #fuzzy查詢 #fuzzy查詢 POST /sms_logs_index/sms_logs_type/_search { "query": { "fuzzy": { "corpName": { "value": "盒馬生鮮", "prefix_length": 2 # 指定前幾個字符要嚴格匹配 } } } } #不穩定，查詢字段差太多也可能查不到 ~~~ ~~~java // java 實現 @Test public void test_query_fuzzy() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.fuzzyQuery("corpName","盒馬生鮮").prefixLength(2)); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } .prefixLength() :指定前幾個字符嚴格匹配 ~~~ #### 4.3.5 wildcard查詢 通配查詢，與mysql中的like查詢是一樣的，可以在查詢時，在字符串中指定通配符*和占位符？ ~~~json #wildcard查詢 POST /sms_logs_index/sms_logs_type/_search { "query": { "wildcard": { "corpName": { "value": "*車" # 可以使用*和？指定通配符和占位符 } } } } ?代表一個占位符 ??代表兩個占位符 ~~~ ~~~java // java代碼 @Test public void test_query_wildcard() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.wildcardQuery("corpName","*車")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } ~~~ #### 4.3.6 range查詢 范圍查詢，只針對數值類型，對某一個Field進行大于或者小于的范圍指定 ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "range": { "relyTotal": { "gte": 0, "lte": 3 } } } } 查詢范圍:[gte,lte] 查詢范圍：(gt,lt) ~~~ ~~~java //java代碼 @Test public void test_query_range() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.rangeQuery("fee").lt(5).gt(2)); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } ~~~ #### 4.3.7 regexp查詢 正則查詢，通過你編寫的正則表達式去匹配內容 PS: prefix,fuzzy,wildcar和regexp查詢效率相對比較低,在對效率要求比較高時，避免去使用 ~~~json POST /sms_logs_index/sms_logs_type/_search { "query": { "regexp": { "moible": "109[0-8]{7}" # 匹配的正則規則 } } } ~~~ ~~~java //java 代碼 @Test public void test_query_regexp() throws IOException { SearchRequest request = new SearchRequest(index); request.types(type); SearchSourceBuilder builder = new SearchSourceBuilder(); builder.query(QueryBuilders.regexpQuery("moible","106[0-9]{8}")); request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request, RequestOptions.DEFAULT); for (SearchHit hit : response.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } } ~~~ ### 4.4 深分頁Scroll ES對from+size有限制，from和size兩者之和不能超過1w 原理： ~~~html from+size ES查詢數據的方式： 1 先將用戶指定的關鍵詞進行分詞處理 2 將分詞去詞庫中進行檢索，得到多個文檔的id 3 去各個分片中拉去指定的數據耗時 4 根據數據的得分進行排序耗時 5 根據from的值，將查詢到的數據舍棄一部分， 6 返回查詢結果 Scroll+size 在ES中查詢方式 1 先將用戶指定的關鍵詞進行分詞處理 2 將分詞去詞庫中進行檢索，得到多個文檔的id 3 將文檔的id存放在一個ES的上下文中，ES內存 4 根據你指定給的size的個數去ES中檢索指定個數的數據，拿完數據的文檔id,會從上下文中移除 5 如果需要下一頁的數據，直接去ES的上下文中，找后續內容 6 循環進行4.5操作 ~~~ 缺點，Scroll是從內存中去拿去數據的，不適合做實時的查詢，拿到的數據不是最新的 ~~~json # 執行scroll查詢，返回第一頁數據，并且將文檔id信息存放在ES的上下文中，指定生存時間 POST /sms_logs_index/sms_logs_type/_search?scroll=1m { "query": { "match_all": {} }, "size": 2, "sort": [ { "fee": { "order": "desc" } } ] } #查詢下一頁的數據 POST /_search/scroll { "scroll_id": "DnF1ZXJ5VGhlbkZldGNoAwAAAAAAACSPFnJjV1pHbENVVGZHMmlQbHVZX1JGdmcAAAAAAAAkkBZyY1daR2xDVVRmRzJpUGx1WV9SRnZnAAAAAAAAJJEWcmNXWkdsQ1VUZkcyaVBsdVlfUkZ2Zw==", "scoll" :"1m" #scorll信息的生存時間 } #刪除scroll在ES中上下文的數據 DELETE /_search/scroll/scrill_id ~~~ ~~~java //java代碼 @Test public void test_query_scroll() throws IOException { // 1 創建SearchRequest SearchRequest request = new SearchRequest(index); request.types(type); // 2 指定scroll信息,生存時間 request.scroll(TimeValue.timeValueMinutes(1L)); // 3 指定查詢條件 SearchSourceBuilder builder = new SearchSourceBuilder(); builder.size(2); builder.sort("fee",SortOrder.DESC); builder.query(QueryBuilders.matchAllQuery()); // 4 獲取返回結果scrollid ,source request.source(builder); RestHighLevelClient client = ESClient.getClient(); SearchResponse response = client.search(request,RequestOptions.DEFAULT); String scrollId = response.getScrollId(); System.out.println(scrollId); while(true){ // 5 循環創建SearchScrollRequest SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId); // 6 指定scrollid生存時間 scrollRequest.scroll(TimeValue.timeValueMinutes(1L)); // 7 執行查詢獲取返回結果 SearchResponse scrollResp = client.scroll(scrollRequest, RequestOptions.DEFAULT); // 8.判斷是否得到數據，輸出 if (scrollResp.getHits().getHits() != null && scrollResp.getHits().getHits().length > 0){ System.out.println("=======下一頁的數據========"); for (SearchHit hit : scrollResp.getHits().getHits()){ System.out.println(hit.getSourceAsMap()); } }else{ // 9。判斷沒有查詢到數據-退出循環 System.out.println("沒得"); break; } } // 10 創建clearScrollRequest ClearScrollRequest clearScrollRequest = new ClearScrollRequest(); // 11 指定scrollid clearScrollRequest.addScrollId(scrollId); // 12 刪除 client.clearScroll(clearScrollRequest,RequestOptions.DEFAULT); } ~~~ ### 4.5 delete-by-query 根據term，match等查詢方式去刪除大量的文檔 如果你需要刪除的內容，是index下的大部分數據，不建議使用，建議逆向操作，創建新的索引，添加需要保留的數據內容 ~~~json POST /sms_logs_index/sms_logs_type/_delete_by_query { "query": { "range": { "relyTotal": { "gte": 2, "lte": 3 } } } } ##中間跟你的查詢條件，查到什么，刪什么t ~~~ ~~~java public class test_sms_search2 { String index = "sms_logs_index"; String type = "sms_logs_type"; @Test public void test_query_fuzzy() throws IOException { DeleteByQueryRequest request = new DeleteByQueryRequest(index); request.types(type); request.setQuery(QueryBuilders.rangeQuery("relyTotal").gt("2").lt("3")); RestHighLevelClient client = ESClient.getClient(); BulkByScrollResponse response = client.deleteByQuery(request, RequestOptions.DEFAULT); System.out.println(response.toString()); } } ~~~ ### 4.6 復合查詢 #### 4.6. 1 bool查詢 復合過濾器，可以將多個查詢條件以一定的邏輯組合在一起，and or - must : 所有的條件，用must組合在一起，表示AND - must_not:將must_not中的條件，全部不能匹配，表示not的意思，不能匹配該查詢條件 - should: 所有條件，用should組合在一起，表示or的意思，文檔必須匹配一個或者多個查詢條件 - filter: 過濾器，文檔必須匹配該過濾條件，跟must子句的唯一區別是，filter不影響查詢的score ~~~json #查詢省份為河北或者河南的 #并且公司名不是河馬生鮮的 #并且smsContext中包含軟件兩個字 POST /sms_logs_index/sms_logs_type/_search { "query": { "bool": { "should": [ { "term": { "province": { "value": "河北" } } }, { "term": { "province": { "value": "河南" } } ], "must_not": [ { "term": { "corpName": { "value": "河馬生鮮" } } } ], "must": [ { "match": { "smsContext": "軟件" } } ] } } } ~~~