数据迁移：关系型数据库 <—-> 大数据平台

通常

• Sqoop工具组件：关系型数据库迁移到大数据平台

• Spark On Mysql：大数据平台反向迁移到关系型数据库(指定数据编码：UTF-8 / GBK)

  本质

• 底层实现是mapreduce，所以sqoop依赖于hadoop，数据是并行导入的

• 相当于对MR的InputFormat和OutputFormat进行定制化设置

  配置

  ```shell

  export HADOOP_COMMON_HOME=

  export HADOOP_COMMON_HOME=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/

Set path to where hadoop-*-core.jar is available

export HADOOP_MAPRED_HOME=

export HADOOP_MAPRED_HOME=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/

set the path to where bin/hbase is available

export HBASE_HOME=

Set the path to where bin/hive is available

export HIVE_HOME=

export HIVE_HOME=/opt/modules/cdh/hive-0.13.1-cdh5.3.6/

Set the path for where zookeper config dir is

export ZOOCFGDIR=

export ZOOCFGDIR=/opt/modules/cdh/zookeeper-3.4.5-cdh5.3.6/ export ZOOKEEPER_HOME=/opt/modules/cdh/zookeeper-3.4.5-cdh5.3.6/ ```

基本命令

使用说明

sqoop help

查看导入命令

sqoop help import