博客地址：http://www.fanlegefan.com
文章地址：http://www.fanlegefan.com/index.php/2017/07/31/prestokerberos/
presto集群安装
安装包下载地址
https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.181/presto-server-0.181.tar.gz
jdk 版本
目前最新的presto版本是0.181，要求jdk不能低于1.8.0_92
集群节点
| 主机名 | ip | role |
|---|---|---|
| slave1 | 192.168.1.115 | coordinator |
| slave2 | 192.168.1.116 | worker |
主机名slave1和slave2没有任何意义，这是以前安装的虚拟机的遗留问题，请忽视这个细节
在slave1节点上安装coordinator
tar -zxvf presto-server-0.181.tar.gz
cd presto-server-0.181
mkdir etc
cd etc
mkdir catalog
mkdir -p /home/qun/data/presto #这个是数据目录，存放日志以及一些配置文件配置Node Properties
- vi etc/node.properties，内容如下
node.environment=production
node.id=ffffffff-ffff-ffff-ffff-ffffffffffff
node.data-dir=/home/qun/data/presto注意node.id在整个集群必须唯一，值可以随便填，没有固定格式
配置JVM Config
- vi etc/jvm.config,内容如下
-server
-Xmx2G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError因为我用的是虚拟机所以调小了jvm内存，官方设置-Xmx16G
配置Config Properties
- vi etc/config.properties,内容如下
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8999
query.max-memory=2GB
query.max-memory-per-node=1GB
discovery-server.enabled=true
discovery.uri=http://slave1:8999设置log
vi etc/log.properties，内容如下
com.facebook.presto=INFO添加mysql connector
- vi etc/catalog/mysql.properties,内容如下
connector.name=mysql
connection-url=jdbc:mysql://192.168.1.116:3306
connection-user=root
connection-password=123456在slave2节点上安装worker
将slave1上配置好的安装包复制到slave2节点上
scp -r presto-server-0.181 qun@slave2:~/创建数据目录
mkdir -p /home/qun/data/presto修改Config Properties
- vi etc/config.properties,内容如下
coordinator=false
http-server.http.port=8999
query.max-memory=2GB
query.max-memory-per-node=1GB
discovery.uri=http://slave1:8999和coordinator不一样的是coordinator=false,代表这是一个worker节点
配置Node Properties
- vi etc/node.properties，内容如下
node.environment=production
node.id=ffffffff-ffff-ffff-ffff-fffffffffff1
node.data-dir=/home/qun/data/presto和coordinator不一样的是node.id=ffffffff-ffff-ffff-ffff-fffffffffff1，node.id必须集群唯一
presto安装后，包结构如下
[qun@slave2 presto-server-0.181]$ tree -I *.jar
.
|-- bin
| |-- launcher
| |-- launcher.properties
| |-- launcher.py
| `-- procname
| `-- Linux-x86_64
| `-- libprocname.so
|-- etc
| |-- catalog
| | `-- mysql.properties
| |-- config.properties
| |-- jvm.config
| `-- node.properties
|-- lib
|-- NOTICE
|-- plugin
| |-- accumulo
| |-- atop
| |-- blackhole
| |-- cassandra
| |-- example-http
| |-- hive-hadoop2
| |-- jmx
| |-- kafka
| |-- localfile
| |-- memory
| |-- ml
| |-- mongodb
| |-- mysql
| |-- postgresql
| |-- presto-thrift
| |-- raptor
| |-- redis
| |-- resource-group-managers
| |-- sqlserver
| |-- teradata-functions
| `-- tpch
`-- README.txt启动集群
在所有的节点上执行如下命令，一般情况下先启动coordinator,再启动worker
./bin/launcher start日志如下：
[qun@slave2 log]$ ll /home/qun/data/presto/var/log/
总用量 8388
-rw-rw-r-- 1 qun qun 8306295 7月 30 22:40 http-request.log
-rw-r--r--. 1 qun qun 1512 7月 29 21:24 launcher.log
-rw-rw-r-- 1 qun qun 26582 7月 30 17:37 server.logpresto集群集成kerberos安全认证
presto集成kerberos，只需要在coordinator上进行配置，worker和coordinator之间的通行还是采用之前的http方式，客户端(presto-cli,jdbc等)访问coordinator需要通过https并且需要进行kerberos认证；presto集群集成kerberos认证只需要在coordinator 节点上进行修改，worker节点保持不变，配置完后我们将用presto-cli的方式连接presto集群
在coordinator节点上安装kerberos client
yum install krb5-libs.x86_64 krb5-workstation.x86_64 krb5修改/etc/krb5.conf,内容如下
[root@slave1 qun]# vi /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = XIAOMI.PRESTO
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
XIAOMI.PRESTO = {
kdc = xiaobin
admin_server = xiaobin
}
[domain_realm]
.xiaomi.presto = XIAOMI.PRESTO
xiaomi.presto = XIAOMI.PRESTO生成keytab，在slave1节点上执行如下命令，生成qun.keytab
kadmin -p admin/admin -q "addprinc -randkey qun@XIAOMI.PRESTO"
kadmin -p admin/admin -q "addprinc -randkey qun/slave1@XIAOMI.PRESTO"
kadmin -p admin/admin -q "ktadd -k /etc/qun.keytab qun@XIAOMI.PRESTO"
kadmin -p admin/admin -q "ktadd -k /etc/qun.keytab qun/slave1@XIAOMI.PRESTO"生成keystore
需要注意的是alias需要和启动presto的用户名一样
[root@slave1 qun]# keytool -genkeypair -alias qun -keyalg RSA -keystore qunkeystore.jks
输入密钥库口令:
再次输入新口令:
您的名字与姓氏是什么?
[Unknown]: slave1
您的组织单位名称是什么?
[Unknown]:
您的组织名称是什么?
[Unknown]:
您所在的城市或区域名称是什么?
[Unknown]:
您所在的省/市/自治区名称是什么?
[Unknown]:
该单位的双字母国家/地区代码是什么?
[Unknown]:
CN=slave1, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown是否正确?
[否]: y
输入 <qun> 的密钥口令
(如果和密钥库口令相同, 按回车):
再次输入新口令:配置jdk，Java Cryptography Extension Policy Files
wget http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip
将解压的jar放到如下目录中，主要是local_policy.jar和US_export_policy.jar
$JAVA_HOME/jre/lib/security/修改config.properties，内容如下
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8999
query.max-memory=2GB
query.max-memory-per-node=1GB
discovery-server.enabled=true
discovery.uri=http://slave1:8999
http-server.authentication.type=KERBEROS
http.server.authentication.krb5.service-name=qun
http.server.authentication.krb5.keytab=/home/qun/presto-server-0.181/qun.keytab
http.authentication.krb5.config=/home/qun/presto-server-0.181/krb5.conf
http-server.https.enabled=true
http-server.https.port=7778
http-server.https.keystore.path=/home/qun/presto-server-0.181/qunkeystore.jks
http-server.https.keystore.key=keystorepd修改jvm.config，内容如下
-server
-Xmx2G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-Dsun.security.krb5.debug=true
-Dlog.enable-console=true修改完后重启coordinator节点
presto-cli通过kerberos认证连接presto集群
- 下载presto-cli
wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.181/presto-cli-0.181-executable.jar
cp presto-cli-0.181-executable.jar presto-cli
chmod +x presto-cli连接presto集群
- 方式一：这个是debug方式，利于找错误，有大量日志输出，内容已省略
[qun@slave1 ~]$ java -jar presto-cli-0.181-executable.jar \
--server https://slave1:7778 \
--enable-authentication \
--krb5-config-path /etc/krb5.conf \
--krb5-principal qun@XIAOMI.PRESTO \
--krb5-keytab-path /home/qun/presto-server-0.181/qun.keytab \
--krb5-remote-service-name qun \
--keystore-path /home/qun/presto-server-0.181/qunkeystore.jks \
--keystore-password keystorepd
presto> select count(*) from mysql.test.user;
_col0
-------
1
(1 row)
Query 20170730_021003_00003_bpf6r, FINISHED, 1 node
Splits: 18 total, 18 done (100.00%)
0:00 [1 rows, 0B] [3 rows/s, 0B/s]- 方式二
./presto-cli \
--server https://slave1:7778 \
--enable-authentication \
--krb5-config-path /etc/krb5.conf \
--krb5-principal qun@XIAOMI.PRESTO \
--krb5-keytab-path /home/qun/qun.keytab \
--krb5-remote-service-name qun \
--keystore-path /home/qun/qunkeystore.jks \
--keystore-password keystorepd
presto> select count(*) from mysql.test.user;
_col0
-------
1
(1 row)
Query 20170730_021003_00003_bpf6r, FINISHED, 1 node
Splits: 18 total, 18 done (100.00%)
0:00 [1 rows, 0B] [3 rows/s, 0B/s]更换一台新的虚拟机slave3，添加一个新的principal：test1@XIAOMI.PRESTO，生成test1.keytab，然后访问presto集群
kadmin -p admin/admin -q "addprinc -randkey test1@XIAOMI.PRESTO"
kadmin -p admin/admin -q "ktadd -k /etc/test1.keytab test1@XIAOMI.PRESTO"
[test@slave3 ~]$ ./presto-cli \
> --server https://slave1:7778 \
> --enable-authentication \
> --krb5-config-path /etc/krb5.conf \
> --krb5-principal test1@XIAOMI.PRESTO \
> --krb5-keytab-path test1.keytab \
> --krb5-remote-service-name qun \
> --keystore-path qunkeystore.jks \
> --keystore-password keystorepd
presto> show catalogs;
Catalog
---------
mysql
system
(2 rows)
Query 20170801_134406_00005_wa4q4, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]
presto> select count(*) from mysql.test.user;
_col0
-------
1
(1 row)
Query 20170801_134419_00006_wa4q4, FINISHED, 1 node
Splits: 18 total, 18 done (100.00%)
0:00 [1 rows, 0B] [3 rows/s, 0B/s]
常见问题
- javax.net.ssl.SSLPeerUnverifiedException
presto> show catalogs;
Error running command: javax.net.ssl.SSLPeerUnverifiedException: Hostname 192.168.1.116 not verified:
certificate: sha256/i+KNkzrrH/NHzUruc9R+f0a/P8Ql/OhOKh9n3JtL1qg=
DN: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
subjectAltNames: []这种问题由keystore生成的时候host指定错误造成的，您的名字与姓氏是什么?[Unknown]: slave1，slave1即是hostname
2. Authentication failed for token
com.facebook.presto.server.security.SpnegoFilter Authentication failed for token
Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)
解决办法如下，下载JCE
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
解压后替换$JAVA_HOME/jre/lib/security目录下的local_policy.jar，US_export_policy.jar