Spark实战(三)本地连接远程Spark(Python环境)

时间:2024-03-31 16:11:00

一、Python环境准备

   远程服务器上Python版本要与本地想匹配,这里本地使用Anaconda来进行安装,远程服务器之间安装。

wget --no-check-certificate https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz
tar -zxvf Python-3.6.7.tgz 
sudo ./configure --prefix=/usr/local/python3
make & make install
建立软连接
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python

   本地之间下载Anaconda安装包,会自动安装相关环境。

二、配置PyCharm

   首先是tool->Deployment->Configuration中新建一个SFTP

Spark实战(三)本地连接远程Spark(Python环境)

Spark实战(三)本地连接远程Spark(Python环境)
   Mappings中Local path路径正斜杠要修改一下改为反斜杠,否则可能会无法识别导致报错
Spark实战(三)本地连接远程Spark(Python环境)

   接下来在file->setting->project interpreter中新增一个python环境,

Spark实战(三)本地连接远程Spark(Python环境)

Spark实战(三)本地连接远程Spark(Python环境)
    接下来需要在configuration中更改运行参数

Spark实战(三)本地连接远程Spark(Python环境)

Spark实战(三)本地连接远程Spark(Python环境)

Spark实战(三)本地连接远程Spark(Python环境)
   因为是远程环境,Environment Variables中需要指明服务器端python路径,spark路径,java路径。

   在file->setting->project interpreter下载安装py4j,pyspark等包,否则会报错

Spark实战(三)本地连接远程Spark(Python环境)

   以下是配置中遇到的一些常见错误:

Spark实战(三)本地连接远程Spark(Python环境)
   去上面搜索安装pyspark-stubs

Spark实战(三)本地连接远程Spark(Python环境)
    上面错误表示py4j版本不兼容,重新下载想匹配的版本

Spark实战(三)本地连接远程Spark(Python环境)
   spark FileNotFoundError: [WinError 2] 系统找不到指定的文件,需要查询edit configuration中python interpreter是否选择正确

   python3安装完成后默认没有安装pip,需要自己去安装,否则会出先pip未找到的错误

备份原来的源:输入命令sudo cp /etc/apt/sources.list /etc/apt/sources_init.list
输入命令sudo vi /etc/apt/sources.list
添加如下
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse 

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo apt-get install python-pip