1.禁用Oozie Hive Action
功能描述:
oozie取消了Hive actions 使用Hive2 actions替代
功能实现:
遇到问题:
2.Oozie支持Spark2
功能描述:
Oozie提供了对spark2的支持。
前提:
Spark2 must be installed on the node where the Oozie server is installed.
说明:
Oozie支持Spark2作为技术预览提供。
功能实现:
Oozie支持Spark2作为技术预览提供,需要手动进行配置:
配置步骤如下:
1.在Oozie ShareLib目录下创建spark2的ShareLib(oozie sharelib配置目录可在ambari上查看:)
hdfs dfs -mkdir hdfs://ocdp/user/ocdc/share/lib/lib_20181012162801/spark2
2.拷贝spark2的jar到spark2的ShareLib
hdfs dfs -put /usr/hdp/3.0.0.0-1634/spark2/jars/ hdfs://ocdp/user/ocdc/share/lib/lib_20181012162801/spark2
3.拷贝oozie-sharelib-spark jar到spark2的ShareLib
hdfs dfs -cp /user/ocdc/share/lib/lib_20181012162801/spark/oozie-sharelib-spark-4.3.1.3.0.0.0-1634.jar \
hdfs://ocdp/user/ocdc/share/lib/lib_20181012162801/spark2
4.如果需要使用hive,拷贝hive-site.xml到spark2的ShareLib
hdfs dfs -put /usr/hdp/3.0.0.0-1634/hive/conf/hive-site.xml hdfs://ocdp/user/ocdc/share/lib/lib_20181012162801/spark2
5.拷贝Python libraries到spark2的ShareLib
hdfs dfs -put /usr/hdp/3.0.0.0-1634/spark2/python/lib/py \
hdfs://ocdp/user/ocdc/share/lib/lib_20181012162801/spark2
6.运行如下命令更新sharelib
oozie admin -sharelibupdate
7.查看spark2 sharelib是否添加成功
oozie admin -shareliblist spark2
8.运行spark job的时候,在配置文件job.properties中添加如下属性:
oozie.action.sharelib.for.spark=spark2