GCS Filesystem

For Hudi storage on GCS, regional buckets provide an DFS API with strong consistency.

GCS Configs

There are two configurations required for Hudi GCS compatibility:

  • Adding GCS Credentials for Hudi
  • Adding required jars to classpath

GCS Credentials

Add the required configs in your core-site.xml from where Hudi can fetch them. Replace the fs.defaultFS with your GCS bucket name and Hudi should be able to read/write from the bucket.

  1.   <property>
  2.     <name>fs.defaultFS</name>
  3.     <value>gs://hudi-bucket</value>
  4.   </property>
  6.   <property>
  7.     <name>fs.gs.impl</name>
  8.     <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
  9.     <description>The FileSystem for gs: (GCS) uris.</description>
  10.   </property>
  12.   <property>
  13.     <name>fs.AbstractFileSystem.gs.impl</name>
  14.     <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
  15.     <description>The AbstractFileSystem for gs: (GCS) uris.</description>
  16.   </property>
  18.   <property>
  19.     <name>fs.gs.project.id</name>
  20.     <value>GCS_PROJECT_ID</value>
  21.   </property>
  22.   <property>
  23.     <name>google.cloud.auth.service.account.enable</name>
  24.     <value>true</value>
  25.   </property>
  26.   <property>
  27.     <name>google.cloud.auth.service.account.email</name>
  28.     <value>GCS_SERVICE_ACCOUNT_EMAIL</value>
  29.   </property>
  30.   <property>
  31.     <name>google.cloud.auth.service.account.keyfile</name>
  32.     <value>GCS_SERVICE_ACCOUNT_KEYFILE</value>
  33.   </property>

GCS Libs

GCS hadoop libraries to add to our classpath

  • com.google.cloud.bigdataoss:gcs-connector:1.6.0-hadoop2