Hive本地模式安装及遇到的问题和解决方案

时间:2022-10-10 07:18:55

Apache Hive是基于Hadoop的一个数据仓库工具,可以将结构化的数据文件映射为一张数据库表,并提供简单的SQL查询功能,可以将SQL语句转换为MapReduce任务进行运行。 其优点是学习成本低,可以通过类SQL语句快速实现简单的MapReduce统计,不必开发专门的MapReduce应用,十分适合数据仓库的统计分析。

Hive有三种运行模式:
1.内嵌模式:将元数据保存在本地内嵌的Derby数据库中,这得使用Hive最简单的方式,不过使用内嵌模式的话,缺点也比较明显,因为一个内嵌的Derby数据库每次只能访问一个数据文件,这也就意味着不支持多会话连接。这种情况应对本地测试可能都有所不足,仅供初学者熟悉应用Hive;
2.本地模式:这种模式是将元数据库保存在本地的独立数据库中(比如说MySQL),这样就能够支持多会话和多用户连接;
3.远程模式:如果我们的Hive客户端比较多,在每个客户端都安装MySQL服务还是会造成一定的冗余和浪费,这种情况下,就可以更进一步,将MySQL也独立出来,将元数据保存在远端独立的MySQL服务中。
因为只有一台电脑,所以本文主要介绍Hive的本地模式安装。

安装前准备

1.正确安装Hadoop
2.下载文件:
apache-hive-1.1.0-bin.tar.gz:https://hive.apache.org/downloads.html
JDBC:http://download.softagency.net/MySQL/Downloads/Connector-J/

解压文件并配置Hive环境变量

解压文件

tar zxf apache-hive-1.1.0-bin.tar.gz
mv apache-hive-1.1.0-bin /usr/local/
tar zxf mysql-connector-java-5.1.35.tar.gz

配置环境变量vim /etc/profile

export HIVE_HOME=/usr/local/apache-hive-1.1.0-bin
export PATH=${PATH}:${HIVE_HOME}/bin

修改Hive配置

hive-config.sh

#vim /usr/local/apache-hive-1.1.0-bin/bin/hive-config.sh
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HIVE_HOME=/usr/local/apache-hive-1.1.0-bin
export HADOOP_HOME=/usr/local/hadoop-2.6.0

hive-env.sh

cp hive-env.sh.template hive-env.sh

hive-site.xml

#cp hive-default.xml.template hive-site.xml
#vim hive-site.xml
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive</value>
    <description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>数据库用户名</value>
    <description>Username to use against metastore database</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>数据库密码</value>
    <description>password to use against metastore database</description>
</property>

#如果不配置下面的部分会产生错误1.
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>自定义目录</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>自定义目录</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>自定义目录</value>
    <description>Location of Hive run time structured log file</description>
</property>
<property>
   <name>hive.server2.logging.operation.log.location</name>
    <value>自定义目录/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>

hive-log4j.properties

#cp hive-log4j.properties.template hive-log4j.properties
#vim hive-log4j.properties
hive.log.dir=自定义目录/log/

在HDFS上建立/tmp和/user/hive/warehouse目录,并赋予组用户写权限。

HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

Mysql配置

#创建数据库
mysql> create database hive;
#赋予访问权限
mysql> grant all privileges on hive.* to root@localhost identified by '密码' with grant option;
mysql> flush privileges;
#将JDBC复制到Hive库目录用于java程序与mysql的连接
cp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /usr/local/apache-hive-1.1.0-bin/lib/

完成以上操作后,启动Hadoop,再在终端输入hive启动Hive,若Hive成功启动,Hive本地模式安装完成。

错误1.

Logging initialized using configuration in file:/usr/local/apache-hive-1.1.0-bin/conf/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:472)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at org.apache.hadoop.fs.Path.initialize(Path.java:206)
    at org.apache.hadoop.fs.Path.<init>(Path.java:172)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:515)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:458)
    ... 8 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
    at java.net.URI.checkPath(URI.java:1804)
    at java.net.URI.<init>(URI.java:752)
    at org.apache.hadoop.fs.Path.initialize(Path.java:203)
    ... 11 more

参考自:hive shell not opening when I have hive-site.xml

错误2.

Logging initialized using configuration in file:/usr/local/apache-hive-1.1.0-bin/conf/hive-log4j.properties
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.TerminalFactory.create(TerminalFactory.java:101)
    at jline.TerminalFactory.get(TerminalFactory.java:158)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:230)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
    at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
    at org.apache.hadoop.hive.cli.CliDriver.getConsoleReader(CliDriver.java:773)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:715)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

错误原因:

Hive has upgraded to Jline2 but jline 0.94 exists in the Hadoop lib.

解决方案(Hive on Spark Getting Started):

1.Delete jline from the Hadoop lib directory (it's only pulled in transitively from ZooKeeper).
2.export HADOOP_USER_CLASSPATH_FIRST=true