1、配置与安装

  • 下载镜像包：链接，1.99版本即为所说的sqoop2，由于使用了hadoop3，所以只能下载该版本。、
  • 官方指导文档：链接。

  • 复制mysql-connector-java-5.1.47.jar包，至目录/opt/sqoop/server/lib/下

  • 修改conf下的sqoop.properties文件： ```xml

    配置hadoop配置路径

    org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop/etc/hadoop

Authentication configuration

去除这部分配置的注释

org.apache.sqoop.security.authentication.type=SIMPLE org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler org.apache.sqoop.security.authentication.anonymous=true

  2. - 将sqoop配置进系统环境profile
  3. ```xml
  4. #SQOOP_HOME
  5. export SQOOP_HOME=/opt/sqoop
  6. export PATH=$PATH:$SQOOP_HOME/bin
  7. export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra
  8. export CATALINA_BASE=$SQOOP_HOME/server
  9. export LOGDIR=$SQOOP_HOME/logs

  • 修改hadoop中的core-site.xml文件

    1. <!-- 配置用户组，sqoop会用这个用户组来运行 -->
    2. <property>
    3. <name>hadoop.proxyuser.root.hosts</name>
    4. <value>*</value>
    5. </property>
    6. <property>
    7. <name>hadoop.proxyuser.root.groups</name>
    8. <value>*</value>
    9. </property>

  • 配置/etc/hadoop/container-executor.cfg文件 ```shell vim /opt/hadoop/etc/hadoop/container-executor.cfg

修改允许在hadoop系统上执行程序的用户，和上部分的配置应该一致

allowed.system.users=root


- 分发配置
```xml
rsync -rvl /opt/hadoop/etc/hadoop/ root@hadoop101:/opt/hadoop/etc/hadoop/
rsync -rvl /opt/hadoop/etc/hadoop/ root@hadoop103:/opt/hadoop/etc/hadoop/
rsync -rvl /opt/hadoop/etc/hadoop/ root@hadoop104:/opt/hadoop/etc/hadoop/

  • 通过命令验证配置

    [root@hadoop102 sqoop]# sqoop2-tool verify
Setting conf dir: /opt/sqoop/bin/../conf
Sqoop home directory: /opt/sqoop
Sqoop tool executor:
      Version: 1.99.7
      Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
      Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
5    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
      at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
      at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
      at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
      at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
      at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)

  • sqoop检测到了hive和hadoop有jar包冲突，我们将hadoop中的slf4j-log4j12-1.7.25.jar复制进hive的lib目录下即可。

    [root@hadoop102 sqoop]# sqoop2-tool verify
Setting conf dir: /opt/sqoop/bin/../conf
Sqoop home directory: /opt/sqoop
Sqoop tool executor:
      Version: 1.99.7
      Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
      Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
4    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
      at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
      at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
      at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
      at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
      at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
      at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)

  • 安装hive时也出过这个错，很明显是guava.jar这个包的问题，同样的，从hadoop中替换到sqoop中

    # 可以发现有三个jar包需要替换
[root@hadoop102 conf]# find /opt/sqoop/ -name guava*.jar
/opt/sqoop/server/lib/guava-11.0.2.jar
/opt/sqoop/shell/lib/guava-11.0.2.jar
/opt/sqoop/tools/lib/guava-11.0.2.jar

  • 再次检测，hive中的derby包和sqoop冲突，拷贝高版本 ```shell Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.services.locks is sealed

cp /opt/hive/lib/derby-10.14.1.0.jar /opt/sqoop/server/lib/ rm derby-10.8.2.2.jar


- 再次检测，成功，所以大部分问题都是hive和sqoop包版本冲突导致的。
```shell
[root@hadoop102 lib]# sqoop2-tool verify
Setting conf dir: /opt/sqoop/bin/../conf
Sqoop home directory: /opt/sqoop
Sqoop tool executor:
        Version: 1.99.7
        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
4    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
2020-04-16T19:27:45,398 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.
  • 运行命令升级下，然后开启服务 ```shell sqoop2-tool upgrade sqoop2-server start

[root@hadoop102 sqoop]# sqoop2-server start Setting conf dir: /opt/sqoop/bin/../conf Sqoop home directory: /opt/sqoop Starting the Sqoop2 server… 0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server. 4 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread 2020-04-20T10:52:33,525 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 2020-04-20T10:52:36,147 INFO [main] org.eclipse.jetty.util.log - Logging initialized @3517ms 2020-04-20T10:52:36,231 INFO [main] org.eclipse.jetty.server.Server - jetty-9.2.13.v20150730 2020-04-20T10:52:36,517 INFO [main] org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager - Updating the current master key for generating delegation tokens 2020-04-20T10:52:36,519 INFO [Thread[Thread-7,5,main]] org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 2020-04-20T10:52:36,519 INFO [Thread[Thread-7,5,main]] org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager - Updating the current master key for generating delegation tokens 2020-04-20T10:52:36,545 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7dbe2ebf{/sqoop,null,AVAILABLE} 2020-04-20T10:52:36,561 INFO [main] org.eclipse.jetty.server.ServerConnector - Started ServerConnector@5db6b845{HTTP/1.1}{0.0.0.0:12000} 2020-04-20T10:52:36,561 INFO [main] org.eclipse.jetty.server.Server - Started @3932ms Sqoop2 server started.

jps验证是否启动

[root@hadoop102 sqoop]# jps 2114 NodeManager 4099 QuorumPeerMain 5476 HMaster 6917 SqoopJettyServer 5676 HRegionServer 6957 Jps 1982 DataNode


- sqoop2将shell客户端后后台服务分开了，现在可以单独启动客户端来操作数据库同步
```shell
sqoop2-shell
  • 常用操作
    #设置错误显示
sqoop:000> set option --name verbose --value true
#连接Sqoop服务端
sqoop:000> set server -host hadoop102
#查看连接状态
sqoop:000> show connector
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+

2、使用

  • 现在通过sqoop2将MySQL数据导入hbase中，首先进入客户端shell中
  • 创建MySQL链接