安装包准备

1）下载tez的依赖包：http://tez.apache.org

2）拷贝apache-tez-0.9.1-bin.tar.gz到node101的/opt/module目录

[twx@node101 module]$ ls
apache-tez-0.9.1-bin.tar.gz


3）解压缩apache-tez-0.9.1-bin.tar.gz

[twx@node101 module]$ tar -zxvf apache-tez-0.9.1-bin.tar.gz

4）修改名称

[twx@node101 module]$ mv apache-tez-0.9.1-bin/ tez-0.9.1


在Hive中配置Tez

1）进入到Hive的配置目录：/opt/module/hive/conf

2）在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置

[twx@node101 conf]$ vim hive-env.sh

添加如下配置:

  1. # Set HADOOP_HOME to point to a specific hadoop install directory
  2. export HADOOP_HOME=/opt/module/hadoop-2.7.2
  4. # Hive Configuration Directory can be controlled by:
  5. export HIVE_CONF_DIR=/opt/module/hive/conf
  7. # Folder containing extra libraries required for hive compilation/execution can be controlled by:
  8. export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录
  9. export TEZ_JARS=""
  10. for jar in `ls $TEZ_HOME |grep jar`; do
  11.     export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
  12. done
  13. for jar in `ls $TEZ_HOME/lib`; do
  14.     export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
  15. done
  17. export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS

3）在hive-site.xml文件中添加如下配置，更改hive计算引擎

  1. <property>
  2.     <name>hive.execution.engine</name>
  3.     <value>tez</value>
  4. </property>

配置Tez

1）在Hive的/opt/module/hive/conf下面创建一个tez-site.xml文件

[twx@node101 conf]$ pwd /opt/module/hive/conf [twx@node101 conf]$ vim tez-site.xml

添加如下内容:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>tez.lib.uris</name>    
  <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
    <name>tez.lib.uris.classpath</name>        
  <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
     <name>tez.use.cluster.hadoop-libs</name>
     <value>true</value>
</property>
<property>
     <name>tez.history.logging.service.class</name>        
     <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>

上传Tez到集群

[twx@node101 conf]$ hadoop fs -mkdir /tez [twx@node101 conf]$ hadoop fs -put /opt/module/tez-0.9.1/ /tez [twx@node101 conf]$ hadoop fs -ls /tez /tez/tez-0.9.1


测试

1）启动Hive

[atguigu@hadoop102 hive]$ bin/hive

2）创建LZO表

hive (default)> create table student( id int, name string);

3）向表中插入数据

hive (default)> insert into student values(1,”zhangsan”);

4）如果没有报错就表示成功了

hive (default)> select * from student; 1 zhangsan


Q&A


1）运行Tez时检查到用过多内存而被NodeManager杀死进程问题:

Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with exitCode: -103 For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt. Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

方案一：或者是关掉虚拟内存检查。我们选这个，修改yarn-site.xml:

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>

重启集群即可