hadoop系列 第二坑： hive hbase关联表问题

时间：2023-11-22 09:45:02

关键词：

hive创建表卡住了

创建hive和hbase关联表卡住了

其实针对这一问题在info级别的日志下是看出哪里有问题的（为什么只能在debug下才能看见呢，不太理解开发者的想法）。

以调试模式进入hive客户端：

hive --hiveconf hive.root.logger=DEBUG,console

执行创建关联表的语句

create table IF NOT EXISTS event_content_temp

(id string,pack string,activity string,title string, sx string, dx string, num bigint)

stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

with serdeproperties ("hbase.columns.mapping" = ":key,cf:pack,cf:activity,cf:title,cf:sx,cf:dx,cf:num")

tblproperties ("hbase.table.name" = "event_content_temp");

显示错误日志：

// :: INFO util.RetryCounter: Sleeping 4000ms before retry #...

// :: INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)

// :: WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:)

        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:)

// :: DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input

java.nio.channels.ClosedChannelException

        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:)

        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:)

        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:)

        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:)

// :: DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown output

java.nio.channels.ClosedChannelException

        at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:)

        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:)

        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:)

        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:)

Interrupting... Be patient, this might take some time.

// :: INFO CliDriver: Interrupting... Be patient, this might take some time.

很显然是zookeeper的问题，可是在ClouderaManager中已经设置了呀，核对之后发现只有Hive Service Configuration Safety Valve for hive-site.xml里加了属性

<property>

  <name>hive.aux.jars.path</name>

  <value>

         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar,

         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hbase.jar,

         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar

  </value>

</property>

<property>

    <name>hbase.zookeeper.quorum</name>

    <value>ip-xxx.cn-north-.compute.internal:,ip-xxxx.cn-north-.compute.internal:,ip-xxxx-.cn-north-.compute.internal:,ip-xxxx-.cn-north-.compute.internal:,ip-xxxx.cn-north-.compute.internal:</value>

</property>

Hive Client Configuration Safety Valve for hive-site.xml里没有添加，这里很好理解，hive服务里配置了hbase的zk，但是hive客户端没有配置（hive-site.xml），建议二者配置一致！

配置完之后，对于关联表的操作都可以顺利完成。

总结：建议hive服务端和客户端的hive-site.xml一样

另外，我建了个QQ群：305994766，希望对大数据、算法研发、系统架构感兴趣的朋友能够加入进来，大家一起学习，共同进步（进群请说明自己的公司-职业-昵称）

相关文章