flume实现kafka到文件测试用例

时间:2023-01-27 14:59:40

kafka 到 file

at2.sources =st2
at2.channels = ct2
at2.sinks = kt2 # For each one of the sources, the type is defined
at2.sources.st2.type = org.apache.flume.source.kafka.KafkaSource
at2.sources.st2.kafka.bootstrap.servers = node0.***:9200,node1.***:9200,node2.***:9200
at2.sources.st2.kafka.topics = flume-test
at2.sources.st2.kafka.consumer.group.id = flume-to-file-test-group-id
# The channel can be defined as follows.
at2.sources.st2.channels = ct2
at2.sources.st2.kafka.batchSize = 100 # Each sink's type must be defined
at2.sinks.kt2.type = file_roll
at2.sinks.kt2.sink.directory = /mnt/home/admin/install/flume/examples/file-dir
at2.sinks.kt2.batchSize = 100
at2.sinks.kt2.sink.rollInterval = 300
#Specify the channel the sink should use
at2.sinks.kt2.channel = ct2 # Each channel's type is defined.
at2.channels.ct2.type = memory
at2.channels.ct2.capacity = 1000
at2.channels.ct2.transactionCapacity = 1000
# Other config values specific to each type of channel(sink or source)
at2.sources.st2.channels = ct2
at2.sinks.kt2.channel = ct2

在该文件所存的目录下,执行以下命令调起:

flume-ng agent -name at2  -conf-file test-kafka-to-file.properties

注:sink.rollInterval=300,表示每隔300秒分割成一个文件,默认为30秒,如果不想分割,可以写为0,数据全部追加在一个文件中。