sqoop job增量导入mysql数据到hive下的表目录

启动sqoop的metastore

安装了sqoop metastore服务的主机上，启动metastore服务。

  1. $ rm -rf metastore/*   # 删除元数据
  2. $ bin/sqoop metastore &
  3. $ jps
  4. 19796 Sqoop

当 last-value 失控的时候，可以删除元数据，重新规划。

定义sqoop的job

定义sqoop job，测试job并执行。

创建密码文件，原因：oozie执行sqoop job任务时，使用—password的job任务，会需要手动输入密码，导致失败。

（猜测，十几次反复测试的结果）

$ echo -n “123456” > 130mysql.pwd

创建hive表，未使用password-file

$ bin/sqoop create-hive-table \ —connect jdbc:mysql://master:3306/mydb \ —username root \ —password 123456 \ —table user \ —hive-table user \ —mysql-delimiters

删除任务

$ bin/sqoop job —delete user_import_incr

使用password-file，该文件需存放在HDFS上

$ ~/Documents/hadoop/bin/hadoop fs -put 130mysql.pwd

定义job

$ bin/sqoop job \ —create user_import_incr \ —meta-connect jdbc:hsqldb:hsql://master:16000/sqoop \ — \ import \ —connect jdbc:mysql://master:3306/mydb \ —username root \ —password-file 130mysql.pwd \ —table user \ —mysql-delimiters \ —incremental append \ —check-column id \ —last-value 0 \ —target-dir /user/hive/warehouse/user \ -m 1

查看job列表

$ bin/sqoop job —list Available jobs: user_import_incr

执行任务

$ bin/sqoop job —exec user_import_incr

使用—password时，运行过程中（第一次执行时）需要输入，import任务里的数据库连接密码

Enter password:

使用—password-file时，不需要输入密码

查询

$ ~/Documents/hive/bin/hive hive> select * from user; OK 1 jack 2 tom 3 white 4 black

查看metastore

$ cat metastore/shared.db.log | grep last INSERT INTO SQOOP_SESSIONS VALUES(‘user_import_incr’,’incremental.last.value’,’0’,’SqoopOptions’) DELETE FROM SQOOP_SESSIONS WHERE JOB_NAME=’user_import_incr’ AND PROPNAME=’incremental.last.value’ AND PROPVAL=’0’ AND PROPCLASS=’SqoopOptions’ INSERT INTO SQOOP_SESSIONS VALUES(‘user_import_incr’,’incremental.last.value’,’4’,’SqoopOptions’)

插入数据

mysql> insert into user values (5, ‘ming’),(6, ‘ning’);

再次执行，不需要输入密码

$ bin/sqoop job —exec user_import_incr $ ~/Documents/hive/bin/hive

查询

hive> select * from user_job; OK 1 jack 2 tom 3 white 4 black 5 ming 6 ning hive> dfs -ls /user/hive/warehouse/user; Found 2 items -rw-r—r— 1 jack supergroup 29 2020-05-16 01:46 /user/hive/warehouse/user/part-m-00000 -rw-r—r— 1 jack supergroup 14 2020-05-16 01:47 /user/hive/warehouse/user/part-m-00001

查看metastore

$ cat shared.db.log | grep last DELETE FROM SQOOP_SESSIONS WHERE JOB_NAME=’user_import_incr’ AND PROPNAME=’incremental.last.value’ AND PROPVAL=’4’ AND PROPCLASS=’SqoopOptions’ INSERT INTO SQOOP_SESSIONS VALUES(‘user_import_incr’,’incremental.last.value’,’6’,’SqoopOptions’)

  1. <a name="gGRyb"></a>
  2. ## 定义oozie的workflow任务
  3. oozie任务调度sqoop增量导入任务
  5. - [https://blog.csdn.net/eason_oracle/article/details/78330755](https://blog.csdn.net/eason_oracle/article/details/78330755)
  6. ```bash
  7. $ mkdir oozie-apps/sqoop-mysql2hdfs-wf/
  8. # 或者
  9. $ cp -r oozie-apps/sqoop-freeform-mysql/ oozie-apps/sqoop-mysql2hdfs-wf
  10. $ ll oozie-apps/sqoop-freeform-mysql/
  11. total 860
  12. -rw-r--r--. 1 jack jack   1024 May 13 20:36 job.properties
  13. -rw-r--r--. 1 jack jack 872303 May 13 22:08 mysql-connector-java-5.1.27-bin.jar
  14. -rw-r--r--. 1 jack jack   2498 May 13 22:09 workflow.xml
  16. [jack@master oozie]$ cp -r examples/apps/sqoop oozie-apps/sqoop-mysql-hive-incr
  17. [jack@master oozie]$ cd oozie-apps/sqoop-mysql-hive-incr/
  18. [jack@master sqoop-mysql-hive-incr]$ ll
  19. total 16
  20. -rw-r--r--. 1 jack jack 1208 May 15 22:35 db.hsqldb.properties
  21. -rw-r--r--. 1 jack jack 1081 May 15 22:35 db.hsqldb.script
  22. -rw-r--r--. 1 jack jack 1002 May 15 22:35 job.properties
  23. -rw-r--r--. 1 jack jack 2085 May 15 22:35 workflow.xml
  24. [jack@master sqoop-mysql-hive-incr]$ rm -rf db.hsqldb.*

需要两个文件：

  • job.properties
  • workflow.xml

    job.properties

    ``` nameNode=hdfs://192.168.32.130:8020 jobTracker=192.168.32.130:8032 queueName=default examplesRoot=oozie-apps

oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr

  1. <a name="u4WR6"></a>
  2. ### workflow.xml
  3. file元素放在archive元素前面。
  4. ```xml
  5. <workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
  6.     <start to="sqoop-node"/>
  8.     <action name="sqoop-node">
  9.         <sqoop xmlns="uri:oozie:sqoop-action:0.2">
  10.             <job-tracker>${jobTracker}</job-tracker>
  11.             <name-node>${nameNode}</name-node>
  12.             <configuration>
  13.                 <property>
  14.                     <name>mapred.job.queue.name</name>
  15.                     <value>${queueName}</value>
  16.                 </property>
  17.             </configuration>
  18.             <arg>job</arg>
  19.             <arg>--meta-connect</arg>
  20.             <arg>jdbc:hsqldb:hsql://192.168.32.130:16000/sqoop</arg>
  21.             <arg>--exec</arg>
  22.             <arg>user_import_incr</arg>
  24.             <archive>mysql-connector-java-5.1.27-bin.jar</archive>
  25.         </sqoop>
  26.         <ok to="end"/>
  27.         <error to="fail"/>
  28.     </action>
  30.     <kill name="fail">
  31.         <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  32.     </kill>
  33.     <end name="end"/>
  34. </workflow-app>

拷贝必要文件

  • mysql-connector-jdbc.jar文件
    1. # jdbc
    2. $ cp ~/Documents/sqoop/lib/mysql-connector-java-5.1.27-bin.jar oozie-apps/sqoop-mysql-hive-incr/

    执行测试

    ```bash $ mysql -uroot -p mysql> insert into user values (7, ‘zhangsan’),(8, ‘lisi’); $ ~/Documents/hive/bin/hive hive> dfs -rm -r -f /user/jack/oozie-apps/sqoop-mysql-hive-incr; Deleted /user/jack/oozie-apps/sqoop-mysql-hive-incr

    上传任务

    $ ~/Documents/hadoop/bin/hadoop fs -put -f oozie-apps/sqoop-mysql-hive-incr/ oozie-apps/ $ export OOZIE_URL=”http://192.168.32.130:11000/oozie“ $ bin/oozie job -config oozie-apps/sqoop-mysql-hive-incr/job.properties -run -verbose -debug -auth kerberos

$ bin/oozie job -info 0000017-200513181559469-oozie-jack-W

$ ~/Document/hive/bin/hive hive> select * from user_job; OK 2 tom 3 white 4 black 5 ming 6 ning 7 zhangsan 8 lisi hive> dfs -ls /user/hive/warehouse/user/;
Found 3 items -rw-r—r— 1 jack supergroup 29 2020-05-15 05:02 /user/hive/warehouse/user/part-m-00000 -rw-r—r— 1 jack supergroup 14 2020-05-15 05:06 /user/hive/warehouse/user/part-m-00001 -rw-r—r— 1 jack supergroup 18 2020-05-15 05:21 /user/hive/warehouse/user/part-m-00002

插入数据，再次测试

mysql> insert into user values (9, ‘wangwu’),(10, ‘zhaoliu’); $ export OOZIE_URL=”http://192.168.32.130:11000/oozie“ $ bin/oozie job -config oozie-apps/sqoop-mysql-hive-incr/job.properties -run -verbose -debug $ bin/oozie job -info 0000023-200513181559469-oozie-jack-W $ ~/Document/hive/bin/hive hive> dfs -ls /user/hive/warehouse/user/; Found 4 items -rw-r—r— 1 jack supergroup 29 2020-05-16 01:46 /user/hive/warehouse/user/part-m-00000 -rw-r—r— 1 jack supergroup 14 2020-05-16 01:47 /user/hive/warehouse/user/part-m-00001 -rw-r—r— 1 jack supergroup 18 2020-05-16 02:17 /user/hive/warehouse/user/part-m-00002 -rw-r—r— 1 jack supergroup 20 2020-05-16 02:21 /user/hive/warehouse/user/part-m-00003 hive> select * from user;
OK 1 jack 2 tom 3 white 4 black 5 ming 6 ning 7 zhangsan 8 lisi 9 wangwu 10 zhaoliu

  1. <a name="qcJuf"></a>
  2. ## 定义oozie的coordination任务
  3. ```bash
  4. $ cp -r examples/apps/cron-schedule/ oozie-apps/sqoop-mysql-hive-incr-cron
  5. # 复制jdbc
  6. $ cp oozie-apps/sqoop-mysql-hive-incr/mysql-connector-java-5.1.27-bin.jar \
  7.   oozie-apps/sqoop-mysql-hive-incr-cron/
  8. $ ll oozie-apps/sqoop-mysql-hive-incr-cron
  9. total 864
  10. -rw-r--r--. 1 jack jack   1591 May 16 02:49 coordinator.xml
  11. -rw-r--r--. 1 jack jack   1285 May 16 02:56 job.properties
  12. -rw-r--r--. 1 jack jack 872303 May 16 03:03 mysql-connector-java-5.1.27-bin.jar
  13. -rw-r--r--. 1 jack jack   1841 May 16 02:46 workflow.xml

job.properties

时间以UTC为准，本身oozie没有做时间配置，时区默认为UTC。
注意：

  • oozie.coord.application.path
  • workflowAppUri ``` nameNode=hdfs://192.168.32.130:8020 jobTracker=192.168.32.130:8032 queueName=default examplesRoot=oozie-apps

oozie.use.system.libpath=true

oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr-cron

start 需设为未来时间，否则任务失败，UTC时间。此时

$ date -u

Sat May 16 09:53:44 UTC 2020

start=2020-05-16T10:00Z end=2020-05-16T010:20Z

workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr-cron

  1. <a name="LAp32"></a>
  2. ### workflow.xml
  3. 内容不变，可直接覆盖。
  4. ```xml
  5. <workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
  6.     <start to="sqoop-node"/>
  8.     <action name="sqoop-node">
  9.         <sqoop xmlns="uri:oozie:sqoop-action:0.2">
  10.             <job-tracker>${jobTracker}</job-tracker>
  11.             <name-node>${nameNode}</name-node>
  12.             <configuration>
  13.                 <property>
  14.                     <name>mapred.job.queue.name</name>
  15.                     <value>${queueName}</value>
  16.                 </property>
  17.             </configuration>
  18.             <arg>job</arg>
  19.             <arg>--meta-connect</arg>
  20.             <arg>jdbc:hsqldb:hsql://192.168.32.130:16000/sqoop</arg>
  21.             <arg>--exec</arg>
  22.             <arg>user_import_incr</arg>
  24.             <archive>mysql-connector-java-5.1.27-bin.jar</archive>
  25.         </sqoop>
  26.         <ok to="end"/>
  27.         <error to="fail"/>
  28.     </action>
  30.     <kill name="fail">
  31.         <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  32.     </kill>
  33.     <end name="end"/>
  34. </workflow-app>

coordinator.xml

5分钟执行一次。

  1. <coordinator-app name="cron-coord" frequency="${coord:minutes(5)}" start="${start}" end="${end}" timezone="UTC"
  2.                  xmlns="uri:oozie:coordinator:0.2">
  3.         <action>
  4.         <workflow>
  5.             <app-path>${workflowAppUri}</app-path>
  6.             <configuration>
  7.                 <property>
  8.                     <name>jobTracker</name>
  9.                     <value>${jobTracker}</value>
  10.                 </property>
  11.                 <property>
  12.                     <name>nameNode</name>
  13.                     <value>${nameNode}</value>
  14.                 </property>
  15.                 <property>
  16.                     <name>queueName</name>
  17.                     <value>${queueName}</value>
  18.                 </property>
  19.             </configuration>
  20.         </workflow>
  21.     </action>
  22. </coordinator-app>

frequency="0/10 * * * *" 每10分钟执行一次 。

测试执行

  1. # 每过5分钟插入数据
  2. $ mysql -uroot -p
  3. mysql> insert into user values (11, 'aaaaa'),(12, 'bbbbb');
  4. mysql> insert into user values (13, 'ccccc'),(14, 'ddddd');
  5. # 上传，发布任务
  6. $ ~/Documents/hadoop/bin/hadoop fs -rm -r -f oozie-apps/sqoop-mysql-hive-incr-cron/
  7. $ ~/Documents/hadoop/bin/hadoop fs -put -f oozie-apps/sqoop-mysql-hive-incr-cron/ oozie-apps/
  8. $ export OOZIE_URL="http://192.168.32.130:11000/oozie"
  9. $ bin/oozie job -config oozie-apps/sqoop-mysql-hive-incr-cron/job.properties -run -verbose -debug
  10. $ bin/oozie job -info 0000068-200513181559469-oozie-jack-C
  11. Job ID : 0000068-200513181559469-oozie-jack-C
  12. ------------------------------------------------------------------------------------------------------------------------------------
  13. Job Name    : cron-coord
  14. App Path    : hdfs://192.168.32.130:8020/user/jack/oozie-apps/sqoop-mysql-hive-incr-cron
  15. Status      : RUNNING
  16. Start Time  : 2020-05-16 10:00 GMT
  17. End Time    : 2020-05-16 10:20 GMT
  18. Pause Time  : -
  19. Concurrency : 1
  20. ------------------------------------------------------------------------------------------------------------------------------------
  21. ID                                         Status    Ext ID                               Err Code  Created              Nominal Time         
  22. 0000068-200513181559469-oozie-jack-C@1     WAITING   -                                    -         2020-05-16 09:55 GMT 2020-05-16 10:00 GMT 
  23. ------------------------------------------------------------------------------------------------------------------------------------
  24. 0000068-200513181559469-oozie-jack-C@2     WAITING   -                                    -         2020-05-16 09:55 GMT 2020-05-16 10:05 GMT 
  25. ------------------------------------------------------------------------------------------------------------------------------------
  26. 0000068-200513181559469-oozie-jack-C@3     WAITING   -                                    -         2020-05-16 09:55 GMT 2020-05-16 10:10 GMT 
  27. ------------------------------------------------------------------------------------------------------------------------------------
  28. 0000068-200513181559469-oozie-jack-C@4     WAITING   -                                    -         2020-05-16 09:55 GMT 2020-05-16 10:15 GMT 
  29. ------------------------------------------------------------------------------------------------------------------------------------
  31. # 杀死进程
  32. $ bin/oozie job --kill 0000068-200513181559469-oozie-jack-C
  33. # oozie日志
  34. $ tail logs/oozie.log

Hive脚本分析数据

修改workflow任务支持hive hql脚本分析

job-non-coor.properties

定义一个非周期性地workflow任务，进行流程测试。

  1. nameNode=hdfs://192.168.32.130:8020
  2. jobTracker=192.168.32.130:8032
  3. queueName=default
  4. examplesRoot=oozie-apps
  6. oozie.use.system.libpath=true
  8. oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr-cron

workflow.xml

添加hive action，注意输出目录。

  • hive.metastore.uris - 不加，报错 【Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]】很莫名其妙！示例没问题！


    1. <workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
    2.   <start to="sqoop-node"/>
    4.   <action name="sqoop-node">
    5.       <sqoop xmlns="uri:oozie:sqoop-action:0.2">
    6.           <job-tracker>${jobTracker}</job-tracker>
    7.           <name-node>${nameNode}</name-node>
    8.           <configuration>
    9.               <property>
    10.                   <name>mapred.job.queue.name</name>
    11.                   <value>${queueName}</value>
    12.               </property>
    13.           </configuration>
    14.           <arg>job</arg>
    15.           <arg>--meta-connect</arg>
    16.           <arg>jdbc:hsqldb:hsql://192.168.32.130:16000/sqoop</arg>
    17.           <arg>--exec</arg>
    18.           <arg>user_import_incr</arg>
    20.           <archive>mysql-connector-java-5.1.27-bin.jar</archive>
    21.       </sqoop>
    22.       <ok to="hive-node"/>
    23.       <error to="fail-sqoop-node"/>
    24.   </action>
    26.   <action name="hive-node">
    27.       <hive xmlns="uri:oozie:hive-action:0.2">
    28.           <job-tracker>${jobTracker}</job-tracker>
    29.           <name-node>${nameNode}</name-node>
    30.           <prepare>
    31.               <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/sqoop-mysql-hive-incr-cron"/>
    32.               <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
    33.           </prepare>
    34.           <configuration>
    35.               <property>
    36.                   <name>mapred.job.queue.name</name>
    37.                   <value>${queueName}</value>
    38.               </property>
    39.               <property>
    40.                   <name>hive.metastore.uris</name>
    41.                   <value>thrift://192.168.32.130:9083</value>
    42.               </property>
    43.           </configuration>
    44.           <script>script.q</script>
    45.           <param>OUTPUT=/user/${wf:user()}/${examplesRoot}/output-data/sqoop-mysql-hive-incr-cron</param>
    46.       </hive>
    47.       <ok to="end"/>
    48.       <error to="fail-hive-node"/>
    49.   </action>
    51.   <kill name="fail-sqoop-node">
    52.       <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    53.   </kill>
    54.   <kill name="fail-hive-node">
    55.       <message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    56.   </kill>
    57.   <end name="end"/>
    58. </workflow-app>

    script.q

    1. -- hive> desc user;
    2. -- OK
    3. -- id                      int                                         
    4. -- name                    string
    5. -- 查询所有name长度大于4的记录
    6. INSERT OVERWRITE DIRECTORY '${OUTPUT}' SELECT id FROM default.user WHERE length(name) > 4;

    测试（修改 ${OUTPUT} 进行测试） ```bash $ cp ~/Documents/oozie/oozie-apps/sqoop-mysql-hive-incr-cron/script.q ~/Documents/hive/hql

${OUTPUT} -> /user/jack/oozie-apps/output-data/sqoop-mysql-hive-incr-cron

$ ~/Documents/hive/bin/hive -f ~/Documents/hive/hql/script.q

  1. <a name="JL2Ev"></a>
  2. ### 执行流程测试
  3. ```bash
  4. # 启动hive metastore
  5. $ ~/Documents/hive/bin/hive --service metastore
  6. # 启动sqoop metastore
  7. $ ~/Documents/sqoop/bin/sqoop metastore
  9. # 上传，发布任务
  10. $ ~/Documents/hadoop/bin/hadoop fs -rm -r -f oozie-apps/sqoop-mysql-hive-incr-cron/
  11. $ ~/Documents/hadoop/bin/hadoop fs -put -f oozie-apps/sqoop-mysql-hive-incr-cron/ oozie-apps/
  12. $ export OOZIE_URL="http://192.168.32.130:11000/oozie"
  13. $ bin/oozie job -config oozie-apps/sqoop-mysql-hive-incr-cron/job-non-coor.properties -run -verbose -debug
  14. $ $ bin/oozie job -info 0000105-200513181559469-oozie-jack-W
  15. Job ID : 0000105-200513181559469-oozie-jack-W
  16. ------------------------------------------------------------------------------------------------------------------------------------
  17. Workflow Name : sqoop-wf
  18. App Path      : hdfs://192.168.32.130:8020/user/jack/oozie-apps/sqoop-mysql-hive-incr-cron
  19. Status        : SUCCEEDED
  20. Run           : 0
  21. User          : jack
  22. Group         : -
  23. Created       : 2020-05-18 06:34 GMT
  24. Started       : 2020-05-18 06:34 GMT
  25. Last Modified : 2020-05-18 06:34 GMT
  26. Ended         : 2020-05-18 06:34 GMT
  27. CoordAction ID: -
  29. Actions
  30. ------------------------------------------------------------------------------------------------------------------------------------
  31. ID                                                                            Status    Ext ID                 Ext Status Err Code  
  32. ------------------------------------------------------------------------------------------------------------------------------------
  33. 0000105-200513181559469-oozie-jack-W@:start:                                  OK        -                      OK         -         
  34. ------------------------------------------------------------------------------------------------------------------------------------
  35. 0000105-200513181559469-oozie-jack-W@sqoop-node                               OK        job_1588552715711_0212 SUCCEEDED  -         
  36. ------------------------------------------------------------------------------------------------------------------------------------
  37. 0000105-200513181559469-oozie-jack-W@hive-node                                OK        job_1588552715711_0213 SUCCEEDED  -         
  38. ------------------------------------------------------------------------------------------------------------------------------------
  39. 0000105-200513181559469-oozie-jack-W@end                                      OK        -                      OK         -         
  40. ------------------------------------------------------------------------------------------------------------------------------------
  41. # 查看结果
  42. $ ~/Documents/hadoop/bin/hadoop fs -cat /user/jack/oozie-apps/output-data/sqoop-mysql-hive-incr-cron/*
  43. 3
  44. 4
  45. 7
  46. 9
  47. 10
  48. 11
  49. 12
  50. 13
  51. 14

执行周期流程测试

job.properties

  2. nameNode=hdfs://192.168.32.130:8020
  3. jobTracker=192.168.32.130:8032
  4. queueName=default
  5. examplesRoot=oozie-apps
  7. oozie.use.system.libpath=true
  9. oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr-cron
  11. # start 需设为未来时间，否则任务失败，UTC时间。此时
  12. # $ date -u
  13. # Mon May 18 03:46:36 UTC 2020
  14. start=2020-05-18T03:50Z
  15. end=2020-05-18T04:10Z
  17. workflowAppUri=${nameNode}/user/${user.name}/${examplesRoot}/sqoop-mysql-hive-incr-cron

执行测试

此时，时间已过，仍然直接运行。很莫名其妙！

  1. $ bin/oozie job -config oozie-apps/sqoop-mysql-hive-incr-cron/job.properties -run -verbose -debug
  2. $ date -u
  3. Mon May 18 06:44:12 UTC 2020
  4. $ bin/oozie job -info 0000106-200513181559469-oozie-jack-C
  5. Job ID : 0000106-200513181559469-oozie-jack-C
  6. ------------------------------------------------------------------------------------------------------------------------------------
  7. Job Name    : cron-coord
  8. App Path    : hdfs://192.168.32.130:8020/user/jack/oozie-apps/sqoop-mysql-hive-incr-cron
  9. Status      : RUNNING
  10. Start Time  : 2020-05-18 03:50 GMT
  11. End Time    : 2020-05-18 04:10 GMT
  12. Pause Time  : -
  13. Concurrency : 1
  14. ------------------------------------------------------------------------------------------------------------------------------------
  15. ID                                         Status    Ext ID                               Err Code  Created              Nominal Time         
  16. 0000106-200513181559469-oozie-jack-C@1     RUNNING   0000107-200513181559469-oozie-jack-W -         2020-05-18 06:43 GMT 2020-05-18 03:50 GMT 
  17. ------------------------------------------------------------------------------------------------------------------------------------
  18. 0000106-200513181559469-oozie-jack-C@2     READY     -                                    -         2020-05-18 06:43 GMT 2020-05-18 03:55 GMT 
  19. ------------------------------------------------------------------------------------------------------------------------------------
  20. 0000106-200513181559469-oozie-jack-C@3     READY     -                                    -         2020-05-18 06:43 GMT 2020-05-18 04:00 GMT 
  21. ------------------------------------------------------------------------------------------------------------------------------------
  22. 0000106-200513181559469-oozie-jack-C@4     READY     -                                    -         2020-05-18 06:43 GMT 2020-05-18 04:05 GMT 
  23. ------------------------------------------------------------------------------------------------------------------------------------
  1. # 上传任务，测试，每过5分钟插入2条数据
  2. $ mysql -uroot -p
  3. mysql> insert into user values (15, '222'),(16, '222222');
  4. mysql> insert into user values (17, '333'),(18, '333333');
  5. $ bin/oozie job -info 0000106-200513181559469-oozie-jack-C
  6. ------------------------------------------------------------------------------------------------------------------------------------
  7. ID                                         Status    Ext ID                               Err Code  Created              Nominal Time         
  8. 0000106-200513181559469-oozie-jack-C@1     SUCCEEDED 0000107-200513181559469-oozie-jack-W -         2020-05-18 06:43 GMT 2020-05-18 03:50 GMT 
  9. ------------------------------------------------------------------------------------------------------------------------------------
  10. 0000106-200513181559469-oozie-jack-C@2     SUCCEEDED 0000108-200513181559469-oozie-jack-W -         2020-05-18 06:43 GMT 2020-05-18 03:55 GMT 
  11. ------------------------------------------------------------------------------------------------------------------------------------
  12. 0000106-200513181559469-oozie-jack-C@3     SUCCEEDED 0000109-200513181559469-oozie-jack-W -         2020-05-18 06:43 GMT 2020-05-18 04:00 GMT 
  13. ------------------------------------------------------------------------------------------------------------------------------------
  14. 0000106-200513181559469-oozie-jack-C@4     SUCCEEDED 0000110-200513181559469-oozie-jack-W -         2020-05-18 06:43 GMT 2020-05-18 04:05 GMT 
  15. ------------------------------------------------------------------------------------------------------------------------------------
  16. $ ~/Documents/hadoop/bin/hadoop fs -cat /user/jack/oozie-apps/output-data/sqoop-mysql-hive-incr-cron/*
  17. # 此时多了，16/18
  18. 3
  19. 4
  20. 7
  21. 9
  22. 10
  23. 11
  24. 12
  25. 13
  26. 14
  27. 16
  28. 18