bigdata_hadoop集群配置_内存分配

时间:2024-07-15 18:04:32

haoop集群  做好内存管理跟重要,不然经常会给抛出个 OutMemory   ,内存溢出

以horntonworks给出推荐配置为样本,给出一种常见的Hadoop集群上各组件的内存分配方案。配置时通过 ambari对应修改,或者后台同步修改 。

【样本】

he final calculation is to determine the amount of RAM per container:

RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))

With these calculations, the YARN and MapReduce configurations can be set:

Configuration File Configuration Setting Value Calculation
yarn-site.xml yarn.nodemanager.resource.memory-mb = Containers * RAM-per-Container
yarn-site.xml yarn.scheduler.minimum-allocation-mb = RAM-per-Container
yarn-site.xml yarn.scheduler.maximum-allocation-mb = containers * RAM-per-Container
mapred-site.xml mapreduce.map.memory.mb = RAM-per-Container
mapred-site.xml mapreduce.reduce.memory.mb = 2 * RAM-per-Container
mapred-site.xml mapreduce.map.java.opts = 0.8 * RAM-per-Container
mapred-site.xml mapreduce.reduce.java.opts = 0.8 * 2 * RAM-per-Container
yarn-site.xml (check) yarn.app.mapreduce.am.resource.mb = 2 * RAM-per-Container
yarn-site.xml (check) yarn.app.mapreduce.am.command-opts = 0.8 * 2 * RAM-per-Container

Note: After installation, both yarn-site.xml and mapred-site.xml are located in the /etc/hadoop/conf folder.

Configuration File Configuration Setting Value Calculation M
yarn-site.xml yarn.nodemanager.resource.memory-mb = Containers * RAM-per-Container(54G)
yarn-site.xml yarn.scheduler.minimum-allocation-mb = 2048
yarn-site.xml yarn.scheduler.maximum-allocation-mb = containers * RAM-per-Container (54G)
mapred-site.xml mapreduce.map.memory.mb = 2048
mapred-site.xml mapreduce.reduce.memory.mb = 4096
mapred-site.xml mapreduce.map.java.opts = 1638
mapred-site.xml mapreduce.reduce.java.opts = 3276
yarn-site.xml (check) yarn.app.mapreduce.am.resource.mb = 2048
yarn-site.xml (check) yarn.app.mapreduce.am.command-opts = 3276

【样例1】

Configuration File Configuration Setting Value Calculation
yarn-site.xml yarn.nodemanager.resource.memory-mb = Containers * RAM-per-Container
yarn-site.xml yarn.scheduler.minimum-allocation-mb = RAM-per-Container
yarn-site.xml yarn.scheduler.maximum-allocation-mb = containers * RAM-per-Container
mapred-site.xml mapreduce.map.memory.mb = RAM-per-Container
mapred-site.xml mapreduce.reduce.memory.mb = 2 * RAM-per-Container
mapred-site.xml mapreduce.map.java.opts = 0.8 * RAM-per-Container
mapred-site.xml mapreduce.reduce.java.opts = 0.8 * 2 * RAM-per-Container
yarn-site.xml (check) yarn.app.mapreduce.am.resource.mb = 2 * RAM-per-Container
yarn-site.xml (check) yarn.app.mapreduce.am.command-opts = 0.8 * 2 * RAM-per-Container

【样例2】

方案最右侧一栏是一个8G VM的分配方案,方案预留1-2G的内存给操作系统,分配4G给Yarn/MapReduce,当然也包括了HIVE,剩余的2-3G是在需要使用HBase时预留给HBase的。 参考:http://blog.****.net/bluishglc/article/details/42436321

【备注】

另外自己通过thrift链接时 ,留意自己加载时 重设配置,造成任务类似失败 <set mapreduce.map.java.opts=-Xmx1024m;>   当hive提交任务

eg:select count(*) from test; 没问题,但是  自己平台提交有问题 。仔细比对配置  例如搜索关键词 :memory ,opts

参考:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html