1. KAFKA下载地址:http://kafka.apache.org/downloads
KAFKA-快速上手-官方网站:http://kafka.apache.org/quickstart
2. 下载完成后解压,修改config目录下的配置文件:server.properties 、zookeeper.properties
1)修改server.properties:
------------------------------------------------- Server Basics -------------------------------------------------
设置broker.id,确保每个broker的id是唯一的,可以默认master节点的为0,其他依次为1、2...
------------------------------------------------- Socket Server Settings -------------------------------------------------
设置listeners,listeners = PLAINTEXT://172.16.106.153:9092
------------------------------------------------- Log Basics -------------------------------------------------
设置num.partitions,分区数量与broker数量一致
------------------------------------------------- Zookeeper -------------------------------------------------
设置zookeeper集群地址,zookeeper.connect = xxx.xxx.xxx.xxx:2181,xxx.xxx.xxx.xxx2181 (最重要的)
2)修改zookeeper.properties:
设置连接参数,在最后添加配置
tickTime=2000
initLimit=8
syncLimit=3
设置broker.id的服务器地址,server.X和IP要与broker.id所在的服务器地址一致
server.0=xxx.xxx.xxx.xxx:2888:3888
server.1=xxx.xxx.xxx.xxx:2889:3889
3)添加zookeeper数据目录的ID文件,在zookeeper数据目录(默认dataDir=/tmp/zookeeper)添加myid文件,写入broker.id的值。到/tmp/zookeeper目录下执行:
第一台机器:echo 0 > myid
第二台机器:echo 1 > myid
3. 启动kafka
启动时,先启动zookeeper,再启动kafka;关闭反之,先关闭kafka,再关闭zookeeper
启动zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties &
启动kafka: bin/kafka-server-start.sh config/server.properties &
4. 测试kafka集群是否部署成功
1)第一台机器上创建一个TOPIC
bin/kafka-topics.sh --create --zookeeper xxx.xxx.xxx.xxx:2181 --replication-factor 1 --partitions 1 --topic test
2)查看TOPIC列表,确认是否创建成功
bin/kafka-topics.sh --list --zookeeper xxx.xxx.xxx.xxx:2181
3)创建生成者
bin/kafka-console-producer.sh --broker-list xxx.xxx.xxx.xxx:9092 --topic test
4)在所有服务器上创建消费者
bin/kafka-console-consumer.sh --bootstrap-server xxx.xxx.xxx.xxx:9092 --from-beginning --topic test
5)在第一台机器上输入消息,回车发送,CTRL+C终止,查看其他机器的消费者是否能接受到消息
###那些年踩过的坑.....
1. WARN: Connection to node 1 could not be established. Broker may not be available.
创建消费者时出的错,重启zookeeper和kafka
2. ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 1 larger than available brokers: 0
kafka未启动,先停止kafka和zookeeper,确保进程关闭,然后再重新启动。
bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
3. WARN Error while fetching metadata with correlation id 0
启动生产者进程报错,没有绑定Kafka启动监听的host信息
vim config/server.properties
listeners=PLAINTEXT://localhost:9092 ,若此处配置的IP,则在执行命令时使用IP:bin/kafka-console-consumer.sh --bootstrap-server xxx.xxx.xxx.xxx:9092 --from-beginning --topic test
4. UnknowHostException:主机名:主机名
linux无法解析域名导致,可以将主机名对应的ip写入到hosts文件中保存,或者直接删除 /etc/Hostname 文件
vim /tec/hosts
IP 主机名 (如:172.10.1.1 name)