Flume性能测试报告(翻译Flume官方wiki报告)

时间:2022-08-03 01:46:29

因使用flume的时候总是会对其性能有所调研,网上找的要么就是自测的
这里找到一份官方wiki的测试报告供大家参考


https://cwiki.apache.org/confluence/display/FLUME/Performance+Measurements+-+round+2


测试环境:

以下测试基于单个agent

hadoop集群配置:20-node Hadoop cluster (1 name node and 19 data nodes).

服务器配置: 24 cores – Xeon E5-2640 v2 @ 2.00GHz, 164 GB RAM,  7200 rpm Hard Drive.

1.     File channel with HDFS Sink (Sequence File):

基于1.4版本的flume测试,source为4个exec,channel为file,sink为hdfs

Flume version: 1.4

Source: 4 x Exec Source, 100k batchSize

HDFS Sink Batch size: 500,000

Event Size: 500 byte events.

Channel: File

Events/Sec
Sinks 1 data dirs 2 data dirs 4 data dirs 6 data dirs 8 data dirs 10 data dirs
1 14.3k(7Mb/s)          
2 21.9k          
4   35.8k        
8     72.5k 77k 78.6(37Mb/s) 76.6k
10     58k      
12     49.3k 49k    
 

Measurements were taken to get an idea around the configuration that yields best performance. So took measurements only for all data points in the grid that made sense. For example it was not necessary to take measurements for multiple dataDirs at single sink, as it was evident multiple HDFS sink would better than single sink config.

混合的多sinks要比单sink的效果好

2.     HDFS Sink:

相比1使用了内存channel ,memory channel

Flume version: 1.4

Channel: Memory

Event Size: 500 byte events.

#hdfs sinks

snappy batch

sz:1.2mill

snappy batch

sz:1.4mill

Sequence File

batch sz:1.2mill

 1  34.3k(17Mb/s)  33k  33k
 2

71k

 75k  69k
 4 141k   145k  141k
 8 271k   273k  251k
 12 382k   380k  370k
 16 478k   538k(240M/s)  486k(232M/s)
 

Some simple observations:

  • increasing number of dataDirs helps FC perf even on single disk systems
  • Increasing  number of sinks helps

 提高sink的数量是有显著效果的

3.     Hive Sink:

hive sink ,channel为内存,flume版本为1.5或者1.6

Flume version: 1.5 & 1.6

Channel: Memory

BatchSz:1million

Event Size: 500 byte events.

  Flume 1.5 Flume 1.6
  Events/s Mps Events/s Mps
  1 Sink      
DELIMITED Text 36,885 18 138,461 66
Json 12,735 6    
         
         
  16 sinks(agent maxed out)    
DELIMITED Text 209,600 100 348,214 166
Json 25,751 12 31,135 14
         
 

 

Observation: Feeding JSON data to Hive sink is much slower, potentially due to higher parsing overhead of JSON in part.

发送json数据格式会慢一些,主要是慢在json的解析上

4.     HBase Sink:

Flume version: 1.5

Channel: Memory

Serializer: RegexHbaseEventSerializer

Total Sinks: 1

Event Size(bytes) Batch Sz:1 Batch Sz:100 Batch Sz:1000 Batch Sz:10000
500   11mb/s   11mb/s
1000 0.5bB/s 14/mb/s 22mb/s 27mb/s
 

5.     ASync HBase Sink:

Flume version: 1.5

Channel: Memory

Serializer: SimpleAsyncHbaseEventSerializer

Total Sinks: 1

Event Size(bytes) Batch Sz:1 Batch Sz:100 Batch Sz:1000
500   0.4mb/s 0.5mb/s
1000 0.8mb/s 0.8mb/s 0.9mb/s
 

6.     Kafka Source:

Flume version: 1.6

Channel: Memory

Sink: Null Sink

Event Size: 1000 bytes

Total Sinks: 1

Batch Size

(bytes)

Mb/s
1,000 62
10,000 112
20,000 125
40,000 147
80,000 153

作 者:小闪电

出处:http://www.cnblogs.com/yueyanyu/

本文版权归作者和博客园共有,欢迎转载、交流,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文链接。如果觉得本文对您有益,欢迎点赞、欢迎探讨。本博客来源于互联网的资源,若侵犯到您的权利,请联系博主予以删除。