命令行 · JAVA

[TOC] # 基本shell命令要符合jRuby的語法進入hbase命令行 ~~~ ./hbase shell ~~~ 幫助命令 ~~~ help ~~~ 查看是誰 ~~~ whoami ~~~ 查看狀態(多少個server,master) ~~~ status ~~~ ~~~ status master ~~~ 查看版本 ~~~ version ~~~ # 創建表 create幫助命令 ~~~ create ~~~ ## 指定版本創建表,2個列族,第二個指定版本 ~~~ create 't_user_info', {NAME => 'base_info'}, {NAME => 'extra_info',VERSIONS=>2} ~~~ 還可以指定region的個數和split的切分,在create幫助命令中查看 **還可以指定最小版本** ~~~ create 'mytable', {NAME => 'colfam1',VERSIONS=>5,MIN_VERSIONS=>'1'} ~~~ 如果當前存儲的所有時間版本都早于TTL,那么至少MIN_VERSION個版本會被保留下來.這樣確保你做查詢的時候所有數據早于TTL時還有結果返回 ## 避免熱點,建表方法 ~~~ create 'tb_splits',{NAME=>'cf',VERSION=>3},{SPLITS=>['a','b','c']} --直接創建多個region的table,每個region的startkey和endkey由參數依次指定，第一個region沒有開始startkey，最后一個沒有endkey --rowkey參數的選擇必須有意義，才能實現均勻分布，否則分多個region無意義 --多個region可以分布在多個regionserver上，添加數據的時候數據可以均勻分布在region中，實現負載均衡 ~~~ # 查看表的描述 ~~~ describe 't_user_info' ~~~ (enabled表示可用,NAME列族的信息,DATA_BLOCK_ENCODING塊的編碼,BLOOMFILTER布隆過濾器,REPLICATION_SCOPE對應的副本數,COMPRESSION是否壓縮,MIN_VERSIONS最小的版本,TTL生命周期,KEEP_DELETED_CELLS刪除數據是否保留,BLOCKSIZEblock塊的大小,IN_MEMORY是否在內存,BLOCKCACHE塊的緩存) # 插入信息幫助命令 ~~~ put ~~~ 插入數據,row key是liu-20-001,base_info是列.后面name是表示符,值為liuyifei ~~~ put 't_user_info', 'liu-20-001','base_info:name','liuyifei' ~~~ ~~~ put 't_user_info','liu-20-001','extra_info:boyfriends','jdxia' put 't_user_info','liu-20-001','extra_info:boyfriends','jdxia1' ~~~ 當往這個列族中的某個列插入數據時,這個列族的某個列才會在里面 # 自增計數器可以－1也可以是0 ~~~ #遞增命中步長默認為1 hbase(main):002:0> incr 'counters', '20150101', 'daily:hits', 1 COUNTER VALUE = 1 0 row(s) in 0.3320 seconds #使用了put去修改計數器會導致后面的錯誤原因是'1'會轉換成Bytes.toBytes() hbase(main):020:0> put 'counters' ,'20150102','daily:hits','1' 0 row(s) in 0.0520 seconds hbase(main):021:0> incr 'counters', '20150102', 'daily:hits', 1 ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: Field is not a long, it's 1 bytes wide at org.apache.hadoop.hbase.regionserver.HRegion.getLongValue(HRegion.java:7647) at org.apache.hadoop.hbase.regionserver.HRegion.applyIncrementsToColumnFamily(HRegion.java:7601) at org.apache.hadoop.hbase.regionserver.HRegion.doIncrement(HRegion.java:7480) at org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:7440) at org.apache.hadoop.hbase.regionserver.RSRpcServices.increment(RSRpcServices.java:551) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2227) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33646) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) ~~~ **獲取計數器的值** ~~~ 通過命令行獲取計數器的值 get_counter 'wc', 'apple01', 'cf:hits' ~~~ # 查詢顯示hbase中的表 ~~~ list ~~~ 查看插入的數據,全表掃描 ~~~ scan 't_user_info' ~~~ 根據row key的信息,查詢 ~~~ get 't_user_info' , 'liu-20-001' ~~~ 根據row key,和列族的標示符查找 ~~~ get 't_user_info' , 'liu-20-001', 'base_info:name' ~~~ 根據版本查找 ~~~ get 't_user_info', 'liu-20-001',{COLUMN=>'extra_info:boyfriends',VERSIONS=>4} ~~~ ## filter 創建表 ~~~ create 'test1', 'lf', 'sf' lf: column family of LONG values (binary value) -- sf: column family of STRING values ~~~ 導入數據 ~~~ put 'test1', 'user1|ts1', 'sf:c1', 'sku1' put 'test1', 'user1|ts2', 'sf:c1', 'sku188' put 'test1', 'user1|ts3', 'sf:s1', 'sku123' put 'test1', 'user2|ts4', 'sf:c1', 'sku2' put 'test1', 'user2|ts5', 'sf:c2', 'sku288' put 'test1', 'user2|ts6', 'sf:s1', 'sku222' ~~~ 一個用戶（userX），在什么時間（tsX），作為rowkey 對什么產品（value：skuXXX），做了什么操作作為列名，比如，c1: click from homepage; c2: click from ad; s1: search from homepage; b1: buy 查詢案例誰的值=sku188 ~~~ scan 'test1', FILTER=>"ValueFilter(=,'binary:sku188')" ROW COLUMN+CELL user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 ~~~ 誰的值包含88 ~~~ scan 'test1', FILTER=>"ValueFilter(=,'substring:88')" ROW COLUMN+CELL user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user2|ts5 column=sf:c2, timestamp=1409122355030, value=sku288 ~~~ 通過廣告點擊進來的(column為c2)值包含88的用戶 ~~~ scan 'test1', FILTER=>"ColumnPrefixFilter('c2') AND ValueFilter(=,'substring:88')" ROW COLUMN+CELL user2|ts5 column=sf:c2, timestamp=1409122355030, value=sku288 ~~~ 通過搜索進來的(column為s)值包含123或者222的用戶 ~~~ scan 'test1', FILTER=>"ColumnPrefixFilter('s') AND ( ValueFilter(=,'substring:123') OR ValueFilter(=,'substring:222') )" ROW COLUMN+CELL user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 user2|ts6 column=sf:s1, timestamp=1409122355970, value=sku222 ~~~ rowkey為user1開頭的 ~~~ scan 'test1', FILTER => "PrefixFilter ('user1')" ROW COLUMN+CELL user1|ts1 column=sf:c1, timestamp=1409122354868, value=sku1 user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 ~~~ FirstKeyOnlyFilter: 一個rowkey可以有多個version,同一個rowkey的同一個column也會有多個的值, 只拿出key中的第一個column的第一個version KeyOnlyFilter: 只要key,不要value ~~~ scan 'test1', FILTER=>"FirstKeyOnlyFilter() AND ValueFilter(=,'binary:sku188') AND KeyOnlyFilter()" ROW COLUMN+CELL user1|ts2 column=sf:c1, timestamp=1409122354918, value= ~~~ 從user1|ts2開始,找到所有的rowkey以user1開頭的 ~~~ scan 'test1', {STARTROW=>'user1|ts2', FILTER => "PrefixFilter ('user1')"} ROW COLUMN+CELL user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 ~~~ 從user1|ts2開始,找到所有的到rowkey以user2開頭 ~~~ scan 'test1', {STARTROW=>'user1|ts2', STOPROW=>'user2'} ROW COLUMN+CELL user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 ~~~ 查詢rowkey里面包含ts3的 ~~~ import org.apache.hadoop.hbase.filter.CompareFilter import org.apache.hadoop.hbase.filter.SubstringComparator import org.apache.hadoop.hbase.filter.RowFilter scan 'test1', {FILTER => RowFilter.new(CompareFilter::CompareOp.valueOf('EQUAL'), SubstringComparator.new('ts3'))} ROW COLUMN+CELL user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 ~~~ 查詢rowkey里面包含ts的 ~~~ import org.apache.hadoop.hbase.filter.CompareFilter import org.apache.hadoop.hbase.filter.SubstringComparator import org.apache.hadoop.hbase.filter.RowFilter scan 'test1', {FILTER => RowFilter.new(CompareFilter::CompareOp.valueOf('EQUAL'), SubstringComparator.new('ts'))} ROW COLUMN+CELL user1|ts1 column=sf:c1, timestamp=1409122354868, value=sku1 user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 user2|ts4 column=sf:c1, timestamp=1409122354998, value=sku2 user2|ts5 column=sf:c2, timestamp=1409122355030, value=sku288 user2|ts6 column=sf:s1, timestamp=1409122355970, value=sku222 ~~~ 加入一條測試數據 ~~~ put 'test1', 'user2|err', 'sf:s1', 'sku999' ~~~ 查詢rowkey里面以user開頭的，新加入的測試數據并不符合正則表達式的規則，故查詢不出來 ~~~ import org.apache.hadoop.hbase.filter.RegexStringComparator import org.apache.hadoop.hbase.filter.CompareFilter import org.apache.hadoop.hbase.filter.SubstringComparator import org.apache.hadoop.hbase.filter.RowFilter scan 'test1', {FILTER => RowFilter.new(CompareFilter::CompareOp.valueOf('EQUAL'),RegexStringComparator.new('^user\d+\|ts\d+$'))} ROW COLUMN+CELL user1|ts1 column=sf:c1, timestamp=1409122354868, value=sku1 user1|ts2 column=sf:c1, timestamp=1409122354918, value=sku188 user1|ts3 column=sf:s1, timestamp=1409122354954, value=sku123 user2|ts4 column=sf:c1, timestamp=1409122354998, value=sku2 user2|ts5 column=sf:c2, timestamp=1409122355030, value=sku288 user2|ts6 column=sf:s1, timestamp=1409122355970, value=sku222 ~~~ 加入測試數據 ~~~ put 'test1', 'user1|ts9', 'sf:b1', 'sku1' ~~~ b1開頭的列中并且值為sku1的 ~~~ scan 'test1', FILTER=>"ColumnPrefixFilter('b1') AND ValueFilter(=,'binary:sku1')" ROW COLUMN+CELL user1|ts9 column=sf:b1, timestamp=1409124908668, value=sku1 ~~~ SingleColumnValueFilter的使用，b1開頭的列中并且值為sku1的 ~~~ import org.apache.hadoop.hbase.filter.CompareFilter import org.apache.hadoop.hbase.filter.SingleColumnValueFilter import org.apache.hadoop.hbase.filter.SubstringComparator scan 'test1', {COLUMNS => 'sf:b1', FILTER => SingleColumnValueFilter.new(Bytes.toBytes('sf'), Bytes.toBytes('b1'), CompareFilter::CompareOp.valueOf('EQUAL'), Bytes.toBytes('sku1'))} ROW COLUMN+CELL user1|ts9 column=sf:b1, timestamp=1409124908668, value=sku1 ~~~ ## zk的使用 hbase zkcli ~~~ ls / [hbase, zookeeper] ~~~ ~~~ [zk: hadoop000:2181(CONNECTED) 1] ls /hbase [meta-region-server, backup-masters, table, draining, region-in-transition, running, table-lock, master, namespace, hbaseid, online-snapshot, replication, splitWAL, recovering-regions, rs] ~~~ ~~~ [zk: hadoop000:2181(CONNECTED) 2] ls /hbase/table [member, test1, hbase:meta, hbase:namespace] ~~~ ~~~ [zk: hadoop000:2181(CONNECTED) 3] ls /hbase/table/test1 [] ~~~ ~~~ [zk: hadoop000:2181(CONNECTED) 4] get /hbase/table/test1 ?master:60000}l$??lPBUF cZxid = 0x107 ctime = Wed Aug 27 14:52:21 HKT 2014 mZxid = 0x10b mtime = Wed Aug 27 14:52:22 HKT 2014 pZxid = 0x107 cversion = 0 dataVersion = 2 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 31 numChildren = 0 ~~~ ## 顯示當前的所有表 ~~~ list ~~~ ## 統計指定表的記錄數 ~~~ count 'hbase_book' ~~~ ## exist檢查表是否存在 ~~~ exist 'hbase_book' ~~~ ## `is_enable/is_disabled` 檢查表是否啟用或禁用 ~~~ is_enable 'hbase_book' is_disable 'hbase_book' ~~~ # 刪除 `###disable 'user'`(新版本不用) 刪除一個列族： ~~~ alter 'user', NAME => 'f1', METHOD => 'delete' 或 alter 'user', 'delete' => 'f1' ~~~ 添加列族f1同時刪除列族f2 ~~~ alter 'user', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'} ~~~ 將user表的f1列族版本號改為5 ~~~ alter 'people', NAME => 'info', VERSIONS => 5 ~~~ 啟用表 ~~~ enable 'user' ~~~ 刪除t_user_info表,liu-20-001這個row key中,base_info這個列族中標示符為name的值 ~~~ delete 't_user_info', 'liu-20-001','base_info:name' ~~~ 刪除表,要先disable,然后才能刪除 ~~~ disable 't_user_info' drop 't_user_info' ~~~ # 修改表結構首先停用user表（新版本不用） ~~~ disable 'user' ~~~ 添加兩個列族f1和f2 ~~~ alter 'people', NAME => 'f1',VERSIONS => 2 alter 'user', NAME => 'f2' ~~~ 啟用表 ~~~ enable 'user' ~~~ # 刪除表 ~~~ disable 'user' drop 'user' ~~~ # 練習創建user表，包含info、data兩個列族 ~~~ create 'user', 'info', 'data' create 'user', {NAME => 'info', VERSIONS => '3'} ~~~ 向user表中插入信息，row key為rk0001，列族info中添加name列標示符，值為zhangsan ~~~ put 'user', 'rk0001', 'info:name', 'zhangsan' ~~~ 向user表中插入信息，row key為rk0001，列族info中添加gender列標示符，值為female ~~~ put 'user', 'rk0001', 'info:gender', 'female' ~~~ 向user表中插入信息，row key為rk0001，列族info中添加age列標示符，值為20 ~~~ put 'user', 'rk0001', 'info:age', 20 ~~~ 向user表中插入信息，row key為rk0001，列族data中添加pic列標示符，值為picture ~~~ put 'user', 'rk0001', 'data:pic', 'picture' ~~~ 獲取user表中row key為rk0001的所有信息 ~~~ get 'user', 'rk0001' ~~~ 獲取user表中row key為rk0001，info列族的所有信息 ~~~ get 'user', 'rk0001', 'info' ~~~ 獲取user表中row key為rk0001，info列族的name、age列標示符的信息 ~~~ get 'user', 'rk0001', 'info:name', 'info:age' ~~~ 獲取user表中row key為rk0001，info、data列族的信息 ~~~ get 'user', 'rk0001', 'info', 'data' get 'user', 'rk0001', {COLUMN => ['info', 'data']} get 'user', 'rk0001', {COLUMN => ['info:name', 'data:pic']} ~~~ 獲取user表中row key為rk0001，列族為info，版本號最新5個的信息 ~~~ get 'user', 'rk0001', {COLUMN => 'info', VERSIONS => 2} get 'user', 'rk0001', {COLUMN => 'info:name', VERSIONS => 5} get 'user', 'rk0001', {COLUMN => 'info:name', VERSIONS => 5, TIMERANGE => [1392368783980, 1392380169184]} ~~~ 獲取user表中row key為rk0001，cell的值為zhangsan的信息 ~~~ get 'user', 'rk0001', FILTER=>"ValueFilter(=,'binary:zhangsan')" scan 'user', FILTER=>"ValueFilter(=,'binary:zhangsan')" ~~~ 獲取user表中row key為rk0001，列標示符中含有a的信息 ~~~ get 'people', 'rk0001', {FILTER => "(QualifierFilter(=,'substring:a'))"} put 'user', 'rk0002', 'info:name', 'fanbingbing' put 'user', 'rk0002', 'info:gender', 'female' put 'user', 'rk0002', 'info:nationality', '中國' get 'user', 'rk0002', {FILTER => "ValueFilter(=, 'binary:中國')"} ~~~ 查詢user表中的所有信息 ~~~ scan 'user' ~~~ 查詢user表中列族為info的信息 ~~~ scan 'user', {COLUMNS => 'info'} scan 'user', {COLUMNS => 'info', RAW => true, VERSIONS => 5} scan 'persion', {COLUMNS => 'info', RAW => true, VERSIONS => 3} ~~~ 查詢user表中列族為info和data的信息 ~~~ scan 'user', {COLUMNS => ['info', 'data']} scan 'user', {COLUMNS => ['info:name', 'data:pic']} ~~~ 查詢user表中列族為info、列標示符為name的信息 ~~~ scan 'user', {COLUMNS => 'info:name'} ~~~ 查詢user表中列族為info、列標示符為name的信息,并且版本最新的5個 ~~~ scan 'user', {COLUMNS => 'info:name', VERSIONS => 5} ~~~ 查詢user表中列族為info和data且列標示符中含有a字符的信息 ~~~ scan 'user', {COLUMNS => ['info', 'data'], FILTER => "(QualifierFilter(=,'substring:a'))"} ~~~ 查詢user表中列族為info，rk范圍是`[rk0001, rk0003)的數據` ~~~ scan 'people', {COLUMNS => 'info', STARTROW => 'rk0001', ENDROW => 'rk0003'} ~~~ 查詢user表中row key以rk字符開頭的 ~~~ scan 'user',{FILTER=>"PrefixFilter('rk')"} ~~~ 查詢user表中指定范圍的數據 ~~~ scan 'user', {TIMERANGE => [1392368783980, 1392380169184]} ~~~ 刪除數據刪除user表row key為rk0001，列標示符為`info:name的數據` ~~~ delete 'people', 'rk0001', 'info:name' ~~~ 刪除user表row key為rk0001，列標示符為`info:name，timestamp為1392383705316的數據` ~~~ delete 'user', 'rk0001', 'info:name', 1392383705316 ~~~ 清空user表中的數據 ~~~ truncate 'people' ~~~ 修改表結構首先停用user表（新版本不用） ~~~ disable 'user' ~~~ 添加兩個列族f1和f2 ~~~ alter 'people', NAME => 'f1' alter 'user', NAME => 'f2' ~~~ 啟用表 ~~~ enable 'user' ~~~ ###disable 'user'(新版本不用) 刪除一個列族： ~~~ alter 'user', NAME => 'f1', METHOD => 'delete' 或 alter 'user', 'delete' => 'f1' ~~~ 添加列族f1同時刪除列族f2 ~~~ alter 'user', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'} ~~~ 將user表的f1列族版本號改為5 ~~~ alter 'people', NAME => 'info', VERSIONS => 5 ~~~ 啟用表 ~~~ enable 'user' ~~~ 刪除表 ~~~ disable 'user' drop 'user' get 'person', 'rk0001', {FILTER => "ValueFilter(=, 'binary:中國')"} get 'person', 'rk0001', {FILTER => "(QualifierFilter(=,'substring:a'))"} scan 'person', {COLUMNS => 'info:name'} scan 'person', {COLUMNS => ['info', 'data'], FILTER => "(QualifierFilter(=,'substring:a'))"} scan 'person', {COLUMNS => 'info', STARTROW => 'rk0001', ENDROW => 'rk0003'} scan 'person', {COLUMNS => 'info', STARTROW => '20140201', ENDROW => '20140301'} scan 'person', {COLUMNS => 'info:name', TIMERANGE => [1395978233636, 1395987769587]} delete 'person', 'rk0001', 'info:name' alter 'person', NAME => 'ffff' alter 'person', NAME => 'info', VERSIONS => 10 get 'user', 'rk0002', {COLUMN => ['info:name', 'data:pic']} ~~~