CDH6.3.2集成hudi 0.10.0

准备工作

java 8，maven3.4以上

开始编译

从gitee上面拉取对应的分支

  1. git clone -b release-0.10.1 https://gitee.com/apache/Hudi.git
  2. # 编译
  3. mvn clean package -DskipTests -Dspark3 -Dscala-2.12
  1. [INFO] Reactor Summary for Hudi 0.9.0:
  2. [INFO] 
  3. [INFO] Hudi ............................................... SUCCESS [ 59.061 s]
  4. [INFO] hudi-common ........................................ SUCCESS [01:13 min]
  5. [INFO] hudi-timeline-service .............................. SUCCESS [  6.151 s]
  6. [INFO] hudi-client ........................................ SUCCESS [  0.097 s]
  7. [INFO] hudi-client-common ................................. SUCCESS [ 29.714 s]
  8. [INFO] hudi-hadoop-mr ..................................... SUCCESS [ 38.765 s]
  9. [INFO] hudi-spark-client .................................. SUCCESS [ 44.479 s]
  10. [INFO] hudi-sync-common ................................... SUCCESS [  0.956 s]
  11. [INFO] hudi-hive-sync ..................................... SUCCESS [  7.672 s]
  12. [INFO] hudi-spark-datasource .............................. SUCCESS [  0.046 s]
  13. [INFO] hudi-spark-common_2.11 ............................. SUCCESS [ 12.661 s]
  14. [INFO] hudi-spark2_2.11 ................................... SUCCESS [ 11.888 s]
  15. [INFO] hudi-spark_2.11 .................................... SUCCESS [ 39.437 s]
  16. [INFO] hudi-utilities_2.11 ................................ SUCCESS [ 52.633 s]
  17. [INFO] hudi-utilities-bundle_2.11 ......................... SUCCESS [01:34 min]
  18. [INFO] hudi-cli ........................................... SUCCESS [ 10.846 s]
  19. [INFO] hudi-java-client ................................... SUCCESS [  1.805 s]
  20. [INFO] hudi-flink-client .................................. SUCCESS [ 23.037 s]
  21. [INFO] hudi-spark3_2.12 ................................... SUCCESS [ 22.693 s]
  22. [INFO] hudi-dla-sync ...................................... SUCCESS [  3.368 s]
  23. [INFO] hudi-sync .......................................... SUCCESS [  0.038 s]
  24. [INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  6.471 s]
  25. [INFO] hudi-hive-sync-bundle .............................. SUCCESS [  1.736 s]
  26. [INFO] hudi-spark-bundle_2.11 ............................. SUCCESS [ 10.716 s]
  27. [INFO] hudi-presto-bundle ................................. SUCCESS [ 22.618 s]
  28. [INFO] hudi-timeline-server-bundle ........................ SUCCESS [  7.461 s]
  29. [INFO] hudi-hadoop-docker ................................. SUCCESS [  1.406 s]
  30. [INFO] hudi-hadoop-base-docker ............................ SUCCESS [  0.863 s]
  31. [INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  1.103 s]
  32. [INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  1.231 s]
  33. [INFO] hudi-hadoop-history-docker ......................... SUCCESS [  0.891 s]
  34. [INFO] hudi-hadoop-hive-docker ............................ SUCCESS [  4.294 s]
  35. [INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [  0.929 s]
  36. [INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [  0.855 s]
  37. [INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  0.859 s]
  38. [INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [  0.859 s]
  39. [INFO] hudi-hadoop-presto-docker .......................... SUCCESS [  0.885 s]
  40. [INFO] hudi-integ-test .................................... SUCCESS [ 18.117 s]
  41. [INFO] hudi-integ-test-bundle ............................. SUCCESS [02:43 min]
  42. [INFO] hudi-examples ...................................... SUCCESS [  5.054 s]
  43. [INFO] hudi-flink_2.11 .................................... SUCCESS [ 12.870 s]
  44. [INFO] hudi-flink-bundle_2.11 ............................. SUCCESS [ 31.194 s]
  45. [INFO] ------------------------------------------------------------------------
  46. [INFO] BUILD SUCCESS
  47. [INFO] ------------------------------------------------------------------------
  48. [INFO] Total time:  13:49 min
  49. [INFO] Finished at: 2021-09-16T13:59:08+08:00
  50. [INFO] ------------------------------------------------------------------------

编译成功

编译好之后文件目录对应Hudi下的packaging目录

  2. [xxx@xxx Hudi]# cd packaging/
  3. [xxx@xxx packaging]# ll
  4. 总用量 36
  5. drwxr-xr-x 4 root root 4096 9月  16 13:58 hudi-flink-bundle
  6. drwxr-xr-x 4 root root 4096 9月  16 13:54 hudi-hadoop-mr-bundle
  7. drwxr-xr-x 4 root root 4096 9月  16 13:54 hudi-hive-sync-bundle
  8. drwxr-xr-x 4 root root 4096 9月  16 13:56 hudi-integ-test-bundle
  9. drwxr-xr-x 4 root root 4096 9月  16 13:54 hudi-presto-bundle
  10. drwxr-xr-x 4 root root 4096 9月  16 13:54 hudi-spark-bundle
  11. drwxr-xr-x 4 root root 4096 9月  16 13:55 hudi-timeline-server-bundle
  12. drwxr-xr-x 4 root root 4096 9月  16 13:51 hudi-utilities-bundle