前置准备Hadoop集群
Scala IDE eclipse集成环境安装在master服务器上
1.Scala IDE下载
2.启动和设置工作空间
3、添加程序包
(1)PyDev
4.设置字符串替代变量
Window-Perferenes-Run/Debug-String Substitution
SPARK_HOME /usr/loca/spark
HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop
PYSPARK_PYTHON /home/hduser/anaconda2/bin/python
5.PyDev设置Python链接库
Window-Perferences-PyDev-Interpreter-Python Interpreter
选择/usr/bin/python2.7
6.PyDev设置anaconda2链接库路径
Window-Perferences-PyDev-Interpreter-Python Interpreter-Libraries-New Folder
7.PyDev设置Spark Python链接库
Window-Perferences-PyDev-Interpreter-Python Interpreter-Libraries-New Egg/Zip(s)
8、PyDev设置环境变量
Window-Perferences-PyDev-Interpreter-Python Interpreter-Environment
SPARK_HOME ${SPARK_HOME}
HADOOP_CONF_DIR ${HADOOP_CONF_DIR}