Flume 学习笔记之 Flume NG概述及单节点安装

时间:2021-12-03 23:37:52

Flume NG概述:

Flume NG是一个分布式,高可用,可靠的系统,它能将不同的海量数据收集,移动并存储到一个数据存储系统中。轻量,配置简单,适用于各种日志收集,并支持 Failover和负载均衡。其中Agent包含Source,Channel和 Sink,三者组建了一个Agent。三者的职责如下所示:

  • Source:用来消费(收集)数据源到Channel组件中
  • Channel:中转临时存储,保存所有Source组件信息
  • Sink:从Channel中读取,读取成功后会删除Channel中的信息

Flume 学习笔记之 Flume NG概述及单节点安装

单节点安装:

  • 下载 tar:
  • 解压:

     tar -zxvf apache-flume-1.7.0-bin.tar.gz

  • 设置环境变量:

export FLUME_HOME=/usr/local/flume

export PATH=$PATH:$FLUME_HOME/bin

  • 修改flume-env.sh:

export JAVA_HOME=/usr/java/jdk1.7.0_80

  • 测试是否安装成功:

Flume 学习笔记之 Flume NG概述及单节点安装

  • 修改配置文件:
a1.sources = r1
a1.sinks = k1
a1.channels = c1 # Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /home/sky/flume/log_exec_tail # Describe the sink
a1.sinks.k1.type = logger # Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
  • 启动Agent:

flume-ng agent -c conf -f /usr/local/flume/conf/exec_tail.conf -n a1 -Dflume.root.logger=INFO,console

  • 测试Flume是否可以接受到日志更新:

Flume 学习笔记之 Flume NG概述及单节点安装

Flume 学习笔记之 Flume NG概述及单节点安装

测试成功!