1. oozie 调用sql文件的workflow 错误汇总:
1)hive2server密码错误。(有时设置可以无密码,有时需要登陆密码,有时是单独的hive2server密码)
Connecting to jdbc:hive2://spark-02:10000/default
Error: Could not open client transport with JDBC Uri: jdbc:hive2://spark-02:10000/default: Peer indicated failure: Error validating the login (state=08S01,code=0)
No current connection
Error: Could not open client transport with JDBC Uri: jdbc:hive2://spark-02:10000/default: Peer indicated failure: Error validating the login (state=08S01,code=0)
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
2)空格无法识别。
INFO : Table sales_analysis.sa_item_sku_day_sales stats: [numFiles=1, numRows=0, totalSize=98, rawDataSize=0]
Error: Error while compiling statement: FAILED: ParseException line 2:8 cannot recognize input near ',' 'dp_idstring' ',' in column type (state=42000,code=40000)
Closing: 0: jdbc:hive2://spark-02:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
原因:
dp_id与string之间的空格无法识别。
所以多加了几个空格,query执行正常。
3)脚本开头的乱码字符无法识别。
可能是脚本出现了乱码,复制到ue等编辑器下,删除开头试试。
2. oozie 无法调用impala sql
原因貌似是,无法访问文件路径,可能是要文件放在oozie 主机上。
解决办法:
1)把简单的impala sql 放在impala的主机上,然后crontab 调用
2)复杂的impala sql,可以把源表和结果表存成 parquet表,然后hive sql执行后,进行元数据同步
3)kudu表只能用 impala 读取到parquet表后,hive才能访问数据
刷新元数据: invalidate metadata; refresh + 表名
3. oozie 调用shell脚本
sync impala脚本内容
#!/bin/bash
export PYTHON_EGG_CACHE=./myeggs
/usr/bin/impala-shell -q 'invalidate metadata'