Hadoop 2.6.0 完全分布式平台搭建

时间:2024-10-10 17:03:38

一、准备软件环境:

hadoop-2.6.0.tar.gz

CentOS release 6.5

jdk-7u67-linux-x64.tar.gz

网络配置:

master1            192.168.11.24

worker1             192.168.11.25

二、操作步骤:

1. 修改主机名

2. 配置ssh无密码登录节点

3. 安装jdk, hadoop

4. 配置分布式环境

slaves

core-site.xml

hdfs-site.xml

mapred-site.xml 【mapred-site.xml.template】

yarn-site.xml

5.  在master上启动Hadoop

bin/hdfs namenode -format

sbin/start-dfs.sh

sbin/start-yarn.sh

通过jps查看各个节点所启动的进程,首先查看master:master节点启动了NameNode、SecondaryNameNode、ResourceManager进程。

接着查看slave1:启动了DataNode和NodeManager进程。

aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYwAAACwCAIAAACTlM5/AAAYi0lEQVR4nO2dS3bcOJaGuR4OexE94ci2/JLk16kRN5G2y4IetnMRXEOn01kVI9tZtkJyr6K7q2oZPQAJ3gvcCwIRpAIh/d/Jc9JCgAAIkn8ACNyf1cN3l/a/T+t//sd//qUCZdN2q56u3XVbboKbP9+71sN7AEQKAFA0ECkAQNFApAAARQORAgAUDUQKAFA0xYlUY/Dbyi6p2840VWO6tt51UwCoqioqUnX/Y+wN3q512w3VNWZlmpuqNx+rpSldY7tx03NpzCJ63X8VCO2v2840VqlSqduu6GsF9py4SC2nT+JTwGpc5ss86+nT6J/K5BbmPfM+jVlmUClf38asckUKgEXJEik7tOraehhkjXfyuAWOf6kK6TVJ4wcMTz35nreZu87YjP0nwxFubhgMCsiWvK6zD7leL/uIKELkfGlzJ6nbzrSuClKI1n7SHtMwkUrvZ9txbWvEGirp+pLm+F0hQjvUFRWvV7guAMTJHkkN8xz7b/tAjJM0m2G4+bT0ShnR2Ke+n+g5CajbbtWn2P+P3/NNQ1XSlUhb3pjVZL30RO2MM36+rLkJ1G3nnlU+WhTb35gVa/5kf0bS+XcDa752fbu2zh7F8gO0eiPXBQCVDURK+EYWRxhaelVVUZHqq6UiNWiEvdvHgvngSB6i+COOoN5ghOWVpD6vm0336FFi+71i3enn97MnS15G8fraxO1FSqlXvS4AqBQoUuOj6x5OWaToiENf95kewUUXfRYUKaX9c4qUX1JUpMLZXrJQ+SIVq3esDSMpkMIMIuXflOMzr6Wzj4ZJnFqjJlIuvSJTwrBe/jOhWC9Ti4Tznf4oOANBpNT2sxbTaVFuP9tZpvvA0wptJGWa/J8XwumeVG/kugCgkiNS/jetvPCcks4KI8lk/aL/lyvBNMP6TO1+1CfFd8aMCyF8/uYPM5RpYHiIer5ZC8z0DNyRtmO19vNZkRmXpTL7ue36Ymml+vW1NdsZ95YiJdcbuy4AaOxqC4LG+LjhHt6a+AXURSp704NX1C5uHHB7KWszJ5iPqC+SvpkzA7aBIq1eAPIpLiwGAAAoBxApAEDJQKQAAEVz8O7yACIFACgWK1IHECkAQJlApAAAReNE6jcuUu6HZM1dIDFd813S8msQWwCzkHVJlPl9nWjP1KQ7yih/IR8rAPIRRYqEiZDgkfx0zXdJy69RRvT8/L5Ojek6t7ebRskUUf5SPlYA5KGNpBxa4G5GejTGLcVfTd37rvhAVXSjYdfSo0PfpT7IxmhhOi6VPLSKD5TsP1W75Ir+1XdMO4SwUREJy4/7ain9oJav+VJp56v3MwCLMyVS2tdpTnpMpFK/rsV9zJoPFB2eUS8nzXeJhvfWLARZ9nXSfKxcRt9/yg9tY8mjmUG8fN1XS+8HuXylHyI+VrrfFgBLExMpbi2webomUlr+KONDovpAKcKnW5rILgWaZUpQdYoTHrVUIkpA5GOy/IgbhNwPSvlaP0QsYpTyAbgJNJGiAxBKbnpVyc9tLP8EzuMjc5Q3m0hFfaw0RR7SNdcW006VH/HVUoY2cvm5IoXFKbBbtIVzcUifm+4+Fn7yy7jvud0T93kUFYGnj5M/zXdJN6WTfJ10HyvxZB1tZ9d5ailvw/xYxPJVkYr0w/T0jfSD6mMV89sCYGkEkQqG96JX73S65ruk51fwimHPi2ZRRNPjvksRvyfN10nxgdL9tsbz4Crg1cXWyHj5LWll4Ksl9kOkfHW6qvlYwQoK7JDJX/fAbOCNmwBsAETqBsDrBwDYHIgUAKBoHkKkAAAlA5ECABTNw5PLhxApAECxPDy5tDoFkQIAlEgvUicQKQBAkTw6WVuR+pTmJyU7E9Xy3ske6X2cuX5SwRZQ/JQPwN3gkVlbnfqU5idFd0XrAWwjYhBxrp+UVyD0CYA7xGNJpCgR36hpkUowWssVHeJPslqturYlxlFiuEzXzW4mBwC4OR6btR1MKSIVRsCP4W1jmjbda0wfe6ZO7DZ5o7ergMSuORvQ/p8FOHkCAObgyena6tSn9b9S/aSqqopExpNY+t5O0tmFeGKR6ifFovM9kaKHc38ErF4BcCt4erq2OuWJVILf07Sdk+fi6Bs8pi6a+xYiRKR8ByTRcg8jKQD2mKdnV09O149P178TkdKmSHTkQle8qR1tzYc35K9RvHKnYGP5NXO+tUJHsvVVMfFiEgcA2DcOz66sTo0iFfN7Ym8IoOWIHuTeIZP+UxHcMTRz25ElLzIwg/kRALeHo/Orw7Orp6dXv18Fa1LFo71EBgBwezg6vzo8vzo82z+R0sduAIBbxPHFtdWpvRMpAMCd4NnFtdWpzxApAECBPH9/bXXq8zVECgBQHs/fXz97D5ECAJTKiw/XVqf+uP43RAoAUBwvPvy0OgWRAgCUyMsPP61O/e3nv1P8pObymZI8phSCd4NufLK7wLbee6Hy3p0FALvj5cefVqeoSEX8pGbxmapZsMy0THmuBnu2g7NmZg2N6To4YgGQzsuPP19+9EWKErGK2thniibXCdvG1TLpSE0N02F+UuQAFqVj//CGPTaze6G5/ELyrh1bJ7anbjvT9I2m/668KCMS/Di85Nwfakr5+Qe9wRYJdRT6J3JeABSHFamXqkhNWx2MKRnTPTUGUISKFI1f9m0VpNEcHeWx6GTijsC01HsZOp2dNWbUMmZH0x8ht8eW3piure2wkYhUQ/2UufYK3lhafuakxURW7h/tvAAokVdWpD5KIqX5PW3tM0Wfiro1k7M3PiCohVRfByU/qWBY5+QoLlLB6SjCrbWnL70xgyiMtfFDqEjJ7RHzc6OuUdRi/SOeFwBF8urjz1eSSGl+T7P4TLEHJOFpGR88byksZfnZjZiWFqmIePUDsGZwknEDMnUWK7VHye+5CdL8kf6BSIF9wYrUq4/+wrk4C5vLZyomChJ0yuMtuovHan5SfPmLmfPR0ceUSHn1jmcstycQx3F9iq5bTY6k9Pz0vGifx35kgEiBfeHVr4FIaX5Pc/pM5dj7kkXhsUz676Ao1U9KmV6xDRTjwrO398GzsppOX61MQ0uuaUWmYc3pjbFMMya6OeJwsJzfO6+uY0osdUXsvAAojle//rQ69Xfl1z2wX6T8WgrAPgGRuh3AXQvcWnqR+hUiBQAoklGk/hsiBQAoD4gUAKBoIFIAgKKBSAEAigYiBQAoGk2k8nyjtPy6n1RVVf2uwrS9nP5uxvl+aN+u/DEiDwCwCKJI5fpGaflVP5eqqmwwTRvL4GAeTF6o2hxsV35aCCEAYDMmp3tJvlFafl2kel2LqhjJ3LXtEIJHRST0V7Jjoa4zduzWZ5BcB4iyqOVL/lNeOaYhIqX7WwEANqQPMFZFKs03SsuvTPfGIOF0kaqHkFgmUpK/kgu+HUZ1rhLNX0krX/GfonG7bLoX828CAGxGTKRyfaOi+akbQeBaPhFrxuQjMAPwVdBlGGrsRUr3V5LL16xdPP8Aml8pHwCwOZpI5fpGbegzlTOS6rO3TqSm/JU8kdIXj+Tyc0UKi1MALIHzk/p7gp+U5hsVyS/6QI1kihSbXmn+SqpIqf5Kcvmq/xR7dwQ99/17SQQA5SOIVK5vVCx/LDqf+ETFB2A8G/cmd6X3/kotcWPqBaemr1oI/ZUi5av+U8wOy3Bdw3QPgFkRR1IAAFAKECkAQNG8hEgBAErGvs8KIgUAKBSIFACgaF4OLweFSAEASuRl5A3GAACwc4oTqeAte+BGsTtf8epQUA4RkRp2Mt7g7VqPe9TZtu7yoC5UcehW0g3rmV+v+68C7X3LaYEAeWzXD1thnTFmuY3TrzuYjbhILXcxxKeA1bjMl/ksT18f4pPcwu2e+aVCAuXr25jVMiJVbdsPWzHLvZx73cE8ZImUHVq5QBP6vSj7LonpehjNcPXJ93zcHyr0kwrr7TrNpUB6zTobtkTOlzZ3kj5uOewfrf2aX1VOP/fDh9aINVTS9fXfv54gjkI/6+lqP0RK9i61EH7k34TK/TneTeNHC193MA/ZI6lhvGv/7QLiJN8lNb1SRjSDr1MfU0d97ER/KNlPKnANnaw35gMlnS9rbgLUIoKPFsX2R/yq8vqZTbCCCG/t+nZtnXhmWj9H0uV+0EpXxCFyX2nXi4iUr1xj2+a+7mAeNhAp4RtZvJn0m6yKilRfLRUpxdVADQDmEcB8xBHUG/eBityRm0336FFi+yN+VZn97MmSl1G8vjYx+cy0fpbT1X5QsNmtso3T3uh9pRXr1lfZqS583cE8FChS493jWbKE/lCin1RQ6NRIKrros6BIKe2fU6T8kqIiFc72ch5Fb0QjpeeKlBvVNaZrm+HgTUXKGOMr6cLXHczDDCKl+i6p6ewjYkwl1xhx2hT9pLx6+c+EYr0xH6jlREptv+pXldvPwxikGv6cnu7ZpiYucGv9rKXnilTVGGMGa+fxVRmx+0oXqX547jnuLHrdwTy8+PDz5YdEkfK/aeWF55R0VhhJJusm/b9cCZI/lOAnpZhGTdQrH6Keb9YCMz0Dd6TtWK39ql9VZj+3XV8srVS/vrZmO+NOFCm5n8X0SD9E+879lkLyyuerXi+3BSG0MFvuuoPZ6EXqw01vQdAYb7+Sd0ntCfELqIsUfJBBSbz48NP+V8RmTjAfEU/U2GZOAEojIlIAALB7IFIAgKKBSAEAigYiBQAoGogUAKBoXny4hkgBAMpFECm2Ze1GdyuReJgZfhpf7n3CYdBXSj+ltScwLKKRrvOh+SLl7lOL5m/MjZczfV5pPUmy+ztgxXaK9Xr3yZbxkHcK1p/aSGqurf8bOAjtZBNpLuy85vVJ4u5wPCBkvhpEXyQSZ5dkOhjN33Yr06b1zEzlaOelOTREmxMET+nt1OqljU69q2mw1B6yiF/biw/X9r8kkZrLv2mqiVJ4oFgMiXNozSoIN6Hl9M++kcNihqJSu5iED9Yksa/U8yHKGyHWbWeGaFr670rt/wn/Iw3hYeahJ1PFRPL3z7lQiNDPc5WjnRcl9yuQ1jzRP9F6U55eLewmw8ct7r8mnl4kfyF+bc/fXz/PEKl5/JvihLdRxC+JysBkbBoN41UiE9NFSr54bHbGfYgyvkuHrxH7dU1ESu7/eL0qkyI19ZWu5R9DMDcTqU3L0c6L5M+dOLMIoYn+iYhU+lhb6/Z0Hzd3l4f+a1qNhfu15YnULP5NU/gPc8SihJytKElhSl4Uvo44kooXmydSQzh1RfswGrCdfTqLiZQ3IkizZJmhHO28+IfJ5XBrjUg7J+vNWA/QRcovfPK5CP3X4jUW69eWIlJDo+fyb5pgM5EKz/YGREpgNpEav8fc37H+n0ekNIc6rYCp/GlP51zluOKiXZEUPi0Nzafaqdeb0fqCRKoYv7bn76/tf9MiRbuPf8nk+jfFkaZ7NEH2UWJDX6WcqEhtsiYlMKNIeXXp/R+vV0W67w0Zxk/3xWT+xGnaXOUMn4TLBWRdafoie1OY1HZqlyBLYdNFSnsu5hKp+Z73bX27BJHyl+/Ge4EM3Lb1b5IbpQ0LtWEnbVDHuicsx+V1cyk+g8gQKbYMSO8SufX6eYXUfLFr/JOdAu9/3ecrsaPFFyikPVeR/KF5E0n2c89Sjn5ebAV44pSiP/iI7Yz7TKVrlFJxjo8bvcsD/7XYyUr5Z3zet/HtUkdSe0furzYAgP1g30Uq6poEANh/9l2kAAC3HIgUAKBoIFIAgKJ59v76GUQKAFAszy4gUgCAgilOpMiumF03ZaeQCL5dNwWAnWJF6tn76z9KeKVVPe4bT9rxnIl7SeT2aL5FGxUjlVO3nWmy9iqTeFwAbhHPLq7tf6JILadP4tPHalxmEDHLOdWaH9OmpQnlNGaVK1IA3EqOL66OL64SRSrD1yaSrocdDE89G194VbnPIvVG29nbE7GjtvS72Y6wn3Nf50071BXVDxtbE3zCD3F+QACUSZZIVVWOr42eXikjKfvU9xM9JwFatHe0fM2nhoiUr1zurLP9biahhQSTMq2fO/u+8ywZ5AewmC1Sb8QPCIDiGEXqOlWkMiwj9JFHRKT6aklum9maZ7Bo78jIRnm+3Toba//KI9PvZoLRXTIc0VRKP9vE7UWK9hDpMPhog/3h+OLq+OL6+OLqj+t/FSJSo2qM5mf9A9uYrnVuupuKlDGeW8fWfjfbEfRzONtLrsUXqbgx11AbRlKgZI7Pr+xganORSvN78rRA9J1Rl+obY0w/FaSvJYiUr4tUP0wL/PW38LvZDm0kZRpZymOE0z1pih3xAwKgOI7O10fniSKV42sTT2eFkWSybsKmKuPWBK4ncvlqO90WhNCiaBu/my1HIqJIDTPfLUWqtwFKOFkASuXobH10vj6+uEpdk1oY7vJ2J9BFKsnoNlIULLbAbeDobH10fnV8Lo+kcpdEQCb6Zs78QrwxHZy2wC3h8Gx9dL4+Ol9/DkQKAAB2z+HZ5eHZ+ugMIgUAKJKnp5eHZ5cQKQBAoTw1P56eXh5CpAAAZfLE/HhqfhyeXn6+gkgBAMrj8cn3J+bH09PLz1f/hEgBAIrj8cn3J+b7E/Pjdy5S/B2VI+IeTC/4jR4S2/eU+N5QP7Zu5g1U1BmKvpoz42j8xg/Acjw++f7k5PsT8+P39ShSZFu3ED8S5GEboz1PKJrfC3bNkBt6cNrr2rP2arNgG+0917GjIVIALMajd/94fPL98cl3KlIUGpXKI1Tlh5NmYUEaXkFZDzZXuFEH2TZGaraiDL3E/Lad7VAFb5vsk8X8qEg/KL5UAIDNISL1f5JI+VG77tmTh0KhjFGRckU1hgSVJWy39oZh7s+moUZ3NIs8klLyDz5WgwfDhB8WHUSy6V7MlwoAsBkP//rno3f/ePROEilJh+iqTSgEnhmIJlI1NRtPGVVpIqUHMMsipeRn8uTao1nBeOYHNP/K487EHwKwHA//+qfVqU9cpGpr8xEb4wjTPe+xZtpCnnlvlDEdBytP95gjgicpkkip+ZkJaLupSGFxCoAlOHj7p9UpKlIplrKCDVFoqla3hsyLuIgQZ5W8kdTwBx2CBYM+wa9Kz09kR52+kXb6rRF/TAAAzMPB2z8PrEhdDiKVtvA8tR7V473xgJC4ASHWHvJJv8YlNVW0txrz+95S1KlSm06yfjBc1zDdA2BWHrz9dvD2GxMpAAAohwdvvh28/XbwFiIFACgSJ1K/QaQAAAXy4M3XB2++Pnj77bfL/4VIAQCK4/7rrw/efH3wBiIFACgSiBQAoGjuv/5ideq3HxApAEB53Hv95f7rL/dff/kviBQAoECISP1PEBbT73Gk+TV/KOrKlJKe7dy0sKsUAKBQ7v3y5d4vX+5xkaKhHin+UP2Lh4PXjmvpavkJiPvaAQC3FlGkKFREVH8o4eOE9KD8FFy1vZNC6yJUWDEkAKbLNLEDAJSELlJDfBp5wlV/KOHjyXSh/BSoNrY0XK8eY5a5sMJ/DoB9phepX9SRFHd4m1GkhPL1LKLji290Rz6KRkIDAPaIUaS+yyJFrVRUtRgzZotUglWLaNRSVYERlbxcRV0NAAB7hyhS3FqXeaNo/lA2KVGk1PIVRhvfmjn6WmM+kq0viImX0FAAwP6gjKTYGwtoftEfik6u6CFaeqR8DXHnQ9sRq3Sycg5TJwBuDwnTvXLJ/GEQALCH7K9IkU0GWHIC4PayvyIFALgT3Hv95d5riBQAoFTuvf4KkQIAlAtECgBQNPdef7U6pe04BwCAXTKI1FeIFACgRIhIMdM7YvcUuKwEeyQ9qyf+2t+8/CHUkSrbhYq/kXiqeJaQVj4AYGFEkSJBvzxqhftJkR3eLKRP85+azq/QmK7riNtBnoJMxQbWzEyB1QUA2DnaSMpBo3Z5BK/88HsClJVfozFd2w4heFykyKCMlFPT6B1SL7X3pImm6ZtB/1150Ts1LcO+XN2vV8rPP+gNsEgoorAl1aZ2bT18DtEEd5gpkeLKQkZGddsJD49nRJCbX8FGKPdxykSkWLTxWBd3lyHPPx202VDlsQ2N6draBiMTkWpc42hQND0XPnKU8zOnKxZjKLWHNLxPaQxUCtxdYiIl6QpdFQoFJtSc3PwiTJ6cSAUH99k80wWaf+Uxjp6onLV0aunnDmqm1Yn5+dBvFDWtPWGxANxlNJHiX/kiwvQtqjm5+fmRw3Nt2k1FSluccuUM4yD3N/XjU6e9Y3VKfm8RjeaPLJZBpACw3FcWzid/3hJsmqLjotz8/Njp6Rt55n2TPHHxPtKMcX2KL2ZNiJSen7aTTlFjrqQQKQAsgkgF0xD+xAtTE4ugObn5Jciac93/LS6Eq+00XNf4mdHpaO2/sYsU3xtXmWZMdHPEVbhDguT32tl1TFkl6yvfhgsLUuBOI46kwHLkvh0HgDvO/wO1H64RQ0Q/zwAAAABJRU5ErkJggg==" alt="" />

遇到的问题及处理方法:【master节点jps时没有namenode】

重新format一下namenode

6. 关闭Hadoop集群也是在master节点上执行:

 sbin/stop-dfs.sh

 sbin/stop-yarn.sh
Hadoop 2.6.0 完全分布式平台搭建