1、Kafka伪分布安装
下载的地址： http://kafka.apache.org/downloads.html
下载的软件：kafka_2.10-0.8.2.1.tgz
1、编辑文件：config/server.properties
[root@baozi kafka_2.10-0.8.2.1]# pwd
/usr/local/kafka_2.10-0.8.2.1
[root@baozi kafka_2.10-0.8.2.1]# ll
总用量 28
drwxr-xr-x. 3 root root 4096 2月 27 06:17 bin
drwxr-xr-x. 2 root root 4096 6月 20 20:35 config
drwxr-xr-x. 2 root root 4096 2月 27 06:17 libs
-rw-r--r--. 1 root root 11358 2月 27 06:02 LICENSE
-rw-r--r--. 1 root root 162 2月 27 06:02 NOTICE
[root@baozi kafka_2.10-0.8.2.1]#
[root@baozi kafka_2.10-0.8.2.1]# vim config/server.properties
host.name=192.168.1.200
log.dirs=/mykafka/kafka-logs
2、启动zookeeper：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-start.sh config/zookeeper.properties &
[root@baozi kafka_2.10-0.8.2.1]# jps
2028 QuorumPeerMain
2122 Jps
[root@baozi kafka_2.10-0.8.2.1]#
3、后台启动kafka：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
[root@baozi kafka_2.10-0.8.2.1]# jps
2134 Kafka
2222 Jps
2028 QuorumPeerMain
[root@baozi kafka_2.10-0.8.2.1]#
4、查看所有topic列表：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181
//这个是在下面执行完创建topic命令之后输入的：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181
[2015-06-20 20:59:52,034] INFO Accepted socket connection from /192.168.1.200:48535 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:59:52,040] INFO Client attempting to establish new session at /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:59:52,042] INFO Established session 0x14e10fd34b3000b with negotiated timeout 30000 for client /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)
test1
test2
[2015-06-20 20:59:52,236] INFO Processed session termination for sessionid: 0x14e10fd34b3000b (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:59:52,240] INFO Closed socket connection for client /192.168.1.200:48535 which had sessionid 0x14e10fd34b3000b (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#
5、创建topic：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test1
6、describe命令：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181 --topic test1
显示：
[2015-06-20 20:55:07,901] INFO Accepted socket connection from /192.168.1.200:48529 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:55:07,907] INFO Client attempting to establish new session at /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:55:07,909] INFO Established session 0x14e10fd34b30005 with negotiated timeout 30000 for client /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:test1 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test1 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test1 Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: test1 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
[2015-06-20 20:55:08,831] INFO Processed session termination for sessionid: 0x14e10fd34b30005 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:55:08,833] INFO Closed socket connection for client /192.168.1.200:48529 which had sessionid 0x14e10fd34b30005 (org.apache.zookeeper.server.NIOServerCnxn)
7、查看所有topic分区情况：
创建topic ->test2
bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test2
命令：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181
[2015-06-20 20:57:22,513] INFO Accepted socket connection from /192.168.1.200:48533 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:57:22,521] INFO Client attempting to establish new session at /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:57:22,524] INFO Established session 0x14e10fd34b30009 with negotiated timeout 30000 for client /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:test1 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test1 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test1 Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: test1 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic:test2 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test2 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: test2 Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: test2 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
[2015-06-20 20:57:23,678] INFO Processed session termination for sessionid: 0x14e10fd34b30009 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:57:23,682] INFO Closed socket connection for client /192.168.1.200:48533 which had sessionid 0x14e10fd34b30009 (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#
8、删除主题topic：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic test1 --zookeeper 192.168.1.200:2181
查看所有主题：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181
[2015-06-20 21:08:51,864] INFO Accepted socket connection from /192.168.1.200:48538 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:08:51,870] INFO Client attempting to establish new session at /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:08:51,872] INFO Established session 0x14e10fd34b3000e with negotiated timeout 30000 for client /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)
test1 - marked for deletion
test2
[2015-06-20 21:08:52,066] INFO Processed session termination for sessionid: 0x14e10fd34b3000e (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:08:52,071] INFO Closed socket connection for client /192.168.1.200:48538 which had sessionid 0x14e10fd34b3000e (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#
9、进入生产者状态：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2
[2015-06-20 21:11:18,523] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
[root@baozi ~]# jps
2134 Kafka
2780 Jps
2028 QuorumPeerMain
2759 ConsoleProducer
[root@baozi ~]#
10、进入消费者：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning
[root@baozi ~]# jps
2793 ConsoleConsumer
2134 Kafka
2843 Jps
2028 QuorumPeerMain
2759 ConsoleProducer
[root@baozi ~]#
11、在生产者这里写入：
hello world
hadoop hive hbase i am baolibin
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2
[2015-06-20 21:11:18,523] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
[2015-06-20 21:14:05,037] INFO Accepted socket connection from /192.168.1.200:48539 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:14:05,045] INFO Client attempting to establish new session at /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,048] INFO Established session 0x14e10fd34b3000f with negotiated timeout 30000 for client /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,621] INFO Accepted socket connection from /192.168.1.200:48540 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:14:05,622] INFO Client attempting to establish new session at /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,626] INFO Established session 0x14e10fd34b30010 with negotiated timeout 6000 for client /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,858] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2 zxid:0x50 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/ids Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:05,870] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x3 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:06,547] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1b zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners/test2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:06,549] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1c zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,679] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x28 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/0 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,681] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x29 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,747] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x2d zxid:0x61 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,754] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2e zxid:0x62 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,771] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x30 zxid:0x64 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/1 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/1 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,774] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x31 zxid:0x65 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
hello world
hadoop hive hbase i am baolibin
在消费者这里会接收到：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning
hello world
hadoop hive hbase i am baolibin
12、关闭kafka：
关闭broker：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-stop.sh config/server.properties &
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-stop.sh config/server1.properties &
关闭zookeeper：
[root@baozi kafka_2.10-0.8.2.1]# bin/zookeeper-server-stop.sh config/zookeeper.properties >/dev/null 2>&1 &
[root@baozi kafka_2.10-0.8.2.1]# jps
3751 Jps
[root@baozi kafka_2.10-0.8.2.1]#