sqoop 客户端配置

一、配置

  1. 1. 安装 sqoop
  2.   sudo apt-get install sqoop
  4. 2. Sqoop 环境
  6.   vim ~/.bashrc
  7.   # SQOOP
  8.   export SQOOP_HOME=/usr/lib/sqoop
  9.   export SQOOP_CONF_DIR=$SQOOP_HOME/conf
  11. 3. Sqoop 配置
  13.   sudo cp $SQOOP_HOME/conf/sqoop-env-template.sh $SQOOP_HOME/conf/sqoop-env.sh
  15.   # 配置 Hive 环境
  16.   sudo vim $SQOOP_HOME/conf/sqoop-env.sh
  18.   export HIVE_CONF_DIR=$HIVE_HOME/conf
  19.   export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
  21.   # 软链 hive 配置文件
  22.   sudo ln -s $HIVE_HOME/conf/hive-site.xml $SQOOP_HOME/conf/

二、常见问题

  1. 1.出现 org.apache.sqoop.Sqoop 找不到主类
  3.   解决 : 把 sqoop 目录下的 sqoop-1.4.4.jar 拷贝到 hadoop 的 lib 目录下
  4.   cd /opt/cloudera/parcels/CDH/lib/hadoop
  5.   sudo ln -s ../../jars/sqoop-1.4.5-cdh5.3.3.jar ./
  7. 2.mysql 类加载不到
  9.   解决 : 下载 mysql JDBC 放到 hadoop 目录下即可
  10.   cd /opt/cloudera/parcels/CDH/lib/hadoop
  11.   sudo ln -s ../../jars/mysql-connector-java-5.1.31.jar ./
  14. 3. HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce 找不到
  15.   ERROR tool.ImportTool: Imported Failed: Parameter 'directory' is not a directory
  17.   解决 : sudo ln -s /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce /usr/lib/hadoop-mapreduce
  20. 4. Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly
  22.   # 配置 Hive 环境
  23.   sudo vim $SQOOP_HOME/conf/sqoop-env.sh
  25.   export HIVE_CONF_DIR=$HIVE_HOME/conf
  26.   export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
  28.   # 软链 hive 配置文件
  29.   sudo ln -s $HIVE_HOME/conf/hive-site.xml $SQOOP_HOME/conf/