2021-6-7

问题：Streamsets写入hdfs报权限问题：HADOOPFS_ 43- Could not create a file/directory under base directory: ‘org .apache .hadoop. security. AccessControlException: Permission denied:

image.png
原因：HDFS权限无此组目录
解决方式：chown修改组为sdc，重启管道

  1. sudo -u hdfs hdfs dfs -chown -R sdc:hive /xx/xx/xx

2021-07-04

问题：sdc无法写入HDFS

ERROR An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Pipeline ‘RLRULMEIHYBHDFS14167ee73d-081f-4c55-8d16-5e91bc034b4f’ validation error : HADOOPFS_44 - Could not verify the base directory: ‘org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error ProductionPipelineRunnable admin ProductionPipelineRunnable-RLRULMEIHYBHDFS14167ee73d-081f-4c55-8d16-5e91bc034b4f-RL_YS_CHEPB HDFS
image.png

原因：该HDFS是高可用，有两个NN，出现错误时，该pipline配置的连接节点是备用状态。备用的NN不支持写操作。
image.png
image.png
解决方式：将主备且换一下即可。
image.png
image.png

Tips：CDH可以在CM手动且换主备； 或者

  • 先停掉 hadoop-daemon.sh stop zkfc
  • 重启 hadoop-daemon.sh start zkfc

**