我可以使用Presto创建一个蜂房转移吗?

Basic question: Can I use Presto as the basis for my Hive metastore?

基本问题:我能使用Presto作为我的蜂箱转移的基础吗?

I am trying to install presto and hive on an amazon aws and I'm having trouble creating the Hive metastore. Presto seems to be installed and working properly apart from failing to connect to the metastore. The problem is that when I run Hive, it fails to create a metastore.

我试图在亚马逊的aws上安装presto和hive，我在创建蜂窝转移时遇到了麻烦。Presto似乎是安装和正常工作，除了没有连接到转移。问题是，当我运行Hive时，它不能创建一个转移。

When I run Hive I get the following errors:

当我运行Hive时，我得到以下错误:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1449)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
    ... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:358)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 20 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:404)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
    ... 17 more

My conf/hive-site.xml is as follows:

我的conf / hive-site。xml是如下:

<property>
<name>hive.metastore.uris</name>
<value>thrift://ec2..........amazonaws.com:8080</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. </description>

<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:presto://ec2...........amazonaws.com:9083/hive;databaseName=metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>presto-jdbc-0.90.jar</value>
<description>Driver class name for a JDBC metastore</description>

Can I use presto to create my metastore or do I have to use mySQL or derby? If I can, I would greatly appreciate any help I can get solving this.

我可以使用presto创建我的转移或者我必须使用mySQL或derby吗?如果可以的话，我会非常感激能帮我解决这个问题的。

Thanks!

谢谢!

4 个解决方案

#1

In theory you should be able to do this. But Presto needs a backend; which backed in Presto using for your metastore db? Whichever backend you use, it needs to support inserts.

理论上，你应该能够做到这一点。但Presto需要一个后端;哪一个支持你的转移?无论您使用哪个后端，都需要支持插入。

#2

There is a Presto bootstrap action for EMR that should handle all of the setup for you.

对于EMR，有一个Presto引导操作，它应该为您处理所有的设置。

#3

Thanks for the replies so far.

谢谢你的回复。

@Dain Sundstrom: I tried your suggestion of the bootstrap action, but had no real joy.

@Dain Sundstrom:我试过你的建议，但并没有真正的快乐。

Having done more research into the hive structure, I don't think it is possible to use presto as I was trying. From what I understand now, presto uses hive as its database, but hive itself needs an underlying database of mySQL or derby which is where I was going wrong. So my plan is to now install mySQL, link hive to it, and do it that way. Step 2 of that plan is pray it works without too much trouble.

在对蜂箱结构做了更多的研究之后，我不认为有可能在我尝试的时候使用presto。根据我现在所理解的，presto使用hive作为它的数据库，但是hive本身需要一个mySQL或derby的底层数据库，这是我出错的地方。所以我的计划是现在安装mySQL，链接hive，这样做。这个计划的第2步是祈祷它不会有太多的麻烦。

Thanks for your help.

谢谢你的帮助。

#4

You may use the following steps when storing your metastore to MySQL:

您可以在存储转移到MySQL时使用以下步骤:

CREATE DATABASE metastore_db;

CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'hivepassword';

GRANT all on *.* to 'hiveuser'@localhost identified by 'hivepassword';

flush privileges;

Once done add the below to your hive-site.xml in $HIVE_HOME/conf/

完成后，将下面的内容添加到您的网站。xml HIVE_HOME / conf /美元

<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://10.0.0.96/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hiveuser</value>
        <description>user name for connecting to mysql server </description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hivepassword</value>
        <description>password for connecting to mysql server </description>
        </property>

        <property>
        <name>hive.metastore.uris</name>
        <value>thrift://10.0.0.96:9083</value>
        <description>Hive metastore Thrift server</description>
        </property>

Then start the metastore service :

然后开始转移服务:

hive --service metastore &

蜂巢——服务metastore &

Just comment if you want to know more or you get stuck at something.

如果你想知道更多，或者你被某件事卡住了，那就评论一下。

Regards, Devi Prasad.

普拉萨德认为,井斜。

#1