haoop集群 做好内存管理跟重要,不然经常会给抛出个 OutMemory ,内存溢出
以horntonworks给出推荐配置为样本,给出一种常见的Hadoop集群上各组件的内存分配方案。配置时通过 ambari对应修改,或者后台同步修改 。
【样本】
he final calculation is to determine the amount of RAM per container:
RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))
With these calculations, the YARN and MapReduce configurations can be set:
Configuration File | Configuration Setting | Value Calculation |
yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container |
mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
Note: After installation, both yarn-site.xml
and mapred-site.xml
are located in the /etc/hadoop/conf
folder.
Configuration File | Configuration Setting | Value Calculation M |
yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container(54G) |
yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = 2048 |
yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container (54G) |
mapred-site.xml | mapreduce.map.memory.mb | = 2048 |
mapred-site.xml | mapreduce.reduce.memory.mb | = 4096 |
mapred-site.xml | mapreduce.map.java.opts | = 1638 |
mapred-site.xml | mapreduce.reduce.java.opts | = 3276 |
yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2048 |
yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 3276 |
【样例1】
Configuration File | Configuration Setting | Value Calculation |
yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = containers * RAM-per-Container |
mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
【样例2】
方案最右侧一栏是一个8G VM的分配方案,方案预留1-2G的内存给操作系统,分配4G给Yarn/MapReduce,当然也包括了HIVE,剩余的2-3G是在需要使用HBase时预留给HBase的。 参考:http://blog.****.net/bluishglc/article/details/42436321
【备注】
另外自己通过thrift链接时 ,留意自己加载时 重设配置,造成任务类似失败 <set mapreduce.map.java.opts=-Xmx1024m;> 当hive提交任务
eg:select count(*) from test; 没问题,但是 自己平台提交有问题 。仔细比对配置 例如搜索关键词 :memory ,opts
参考:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html