kafka和zookeeper集群安裝部署 · 運維知識記錄

kafka和zookeeper安裝配置 ### 配置java ``` cd /opt/soft tar zxf jdk-8u151-linux-x64.tar.gz -C /opt/ ln -s /opt/jdk1.8.0_151/ /opt/jdk sed -i.bak '$a export JAVA_HOME=/opt/jdk\nexport PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH\nexport CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar' /etc/profile source /etc/profile java -version ``` ### 配置zookeeper ``` cd /opt/soft tar zxf zookeeper-3.4.11.tar.gz -C /opt/ ln -s /opt/zookeeper-3.4.11/ /opt/zookeeper mkdir -p /data/zookeeper cp /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg vim /opt/zookeeper/conf/zoo.cfg grep "^[a-Z]" /opt/zookeeper/conf/zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper clientPort=2181 server.1=192.168.47.70:2888:3888 server.1=192.168.47.71:2888:3888 server.1=192.168.47.72:2888:3888 echo "1" > /data/zookeeper/myid cat /data/zookeeper/myid tree -L 1 /opt/ ``` ### 啟動和關閉zookeeper ``` /opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/bin/zkServer.sh status /opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/bin/zkServer.sh stop ``` ### 發送消息 ``` /opt/zookeeper/bin/zkCli.sh -server 192.168.47.70:2181 [zk: 192.168.47.70:2181(CONNECTED) 0] create /test "hello" Created /test [zk: 192.168.47.70:2181(CONNECTED) 1] ``` ### 接收消息 ``` /opt/zookeeper/bin/zkCli.sh -server 192.168.47.71:2181 [zk: 192.168.47.71:2181(CONNECTED) 0] get /test hello cZxid = 0x100000002 ctime = Mon Mar 12 15:15:52 CST 2018 mZxid = 0x100000002 mtime = Mon Mar 12 15:15:52 CST 2018 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 5 numChildren = 0 [zk: 192.168.47.71:2181(CONNECTED) 1] ``` ### 安裝kafka ``` cd /opt/soft tar zxf kafka_2.11-1.0.0.tgz -C /opt/ ln -s /opt/kafka_2.11-1.0.0/ /opt/kafka mkdir /opt/kafka/logs vim /opt/kafka/config/server.properties ============================================ 21 broker.id=3 31 listeners=PLAINTEXT://192.168.47.72:9092 60 log.dirs=/opt/kafka/logs 103 log.retention.hours=24 123 zookeeper.connect=192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 ============================================ ``` ### 啟動kafka ``` /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties ``` ### 查看日志 ``` tail -f /opt/kafka/logs/server.log ``` ### 驗證進程 ``` /opt/jdk/bin/jps ``` ### 測試創建topic:任意機器創建 ``` /opt/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 --partitions 3 --replication-factor 3 --topic kafkatest ``` ### 測試獲取topic:任意機器查詢 ``` /opt/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 --topic kafkatest ``` ### 測試刪除topic ``` /opt/kafka/bin/kafka-topics.sh --delete --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 --topic kafkatest ``` ### 獲取所有topic ``` /opt/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 ``` ### 測試命令消息發送 ### 創建topic ``` /opt/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 --partitions 3 --replication-factor 3 --topic messagetest ``` ### 發送消息:注意,是9092端口 ``` /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.47.70:9092,192.168.47.71:9092,192.168.47.72:9092 --topic messagetest ``` ### 其他kafka接受消息 ``` /opt/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.47.70:2181,192.168.47.71:2181,192.168.47.72:2181 --topic messagetest --from-beginning ```