1. spark job日志介绍

spark中提供了log4j的方式记录日志。可以在$SPARK_HOME/conf/下，将 log4j.properties.template 文件copy为 log4j.properties 来启用log4j配置。但这个配置为全局配置，不能单独配置某个job的运行日志，但可以使用下面两个方法单独配置。
在Spark的conf目录下，把log4j.properties.template修改为log4j.properties，原文件 log4j.properties 如下

  1. #
  2. # Licensed to the Apache Software Foundation (ASF) under one or more
  3. # contributor license agreements.  See the NOTICE file distributed with
  4. # this work for additional information regarding copyright ownership.
  5. # The ASF licenses this file to You under the Apache License, Version 2.0
  6. # (the "License"); you may not use this file except in compliance with
  7. # the License.  You may obtain a copy of the License at
  8. #
  9. #    http://www.apache.org/licenses/LICENSE-2.0
  10. #
  11. # Unless required by applicable law or agreed to in writing, software
  12. # distributed under the License is distributed on an "AS IS" BASIS,
  13. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  14. # See the License for the specific language governing permissions and
  15. # limitations under the License.
  16. #
  18. # Set everything to be logged to the console
  19. log4j.rootCategory=WARN, console
  20. log4j.appender.console=org.apache.log4j.ConsoleAppender
  21. log4j.appender.console.target=System.err
  22. log4j.appender.console.layout=org.apache.log4j.PatternLayout
  23. log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
  25. # Settings to quiet third party logs that are too verbose
  26. log4j.logger.org.spark-project.jetty=WARN
  27. log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
  28. log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
  29. log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
  30. log4j.logger.org.apache.parquet=ERROR
  31. log4j.logger.parquet=ERROR
  33. # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
  34. log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
  35. log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

2. spark job独立配置的log4j的方法

spark (streaming) job独立配置的log4j的方法，通过查看官方文档，要为应用主程序（即driver端）或执行程序使（即executor端）自定义log4j配置，需要两步就完成了，下面具体说明。
第一步：上传自定义 log4j-driver.properties和log4j-executor.properties 【仔细一看，与上面文件log4j.properties命名的区别】
第二步：添加-Dlog4j的配置。使用 —conf参数。
用于驱动程序(driver端)：spark.driver.extraJavaOptions = -Dlog4j.configuration = <配置文件的位置> 或者 用—driver-java-options参数进行设置
用于执行者executor：spark.executor.extraJavaOptions= -Dlog4j.configuration = <配置文件的位置>

方案一：使用 spark-submit的 —files 参数将自定义的配置文件上传到应用程序的文件列表中。

  1. spark-submit 
  2. --driver-java-options "-Dlog4j.configuration=log4j-driver.properties"
  3. --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j-executor.properties"
  4. --files /home/hadoop/spark-workspace/log4j-driver.properties,/home/hadoop/spark-workspace/log4j-executor.properties
  5. /home/hadoop/spark-workspace/my-spark-etl-assembly-1.0-SNAPSHOT.jar

注意，这里我没有使用spark.driver.extraJavaOptions参数去配置，而是使用spark-submit的—driver-java-options参数进行设置的。

方案二：不使用 spark-submit的 —files 参数上传文件，直接使用文件。

  1. spark-submit
  2. --class com.hm.spark.Application
  3. --master yarn --deploy-mode cluster
  4. --driver-cores 1 --driver-memory 1G
  5. --num-executors 2 --executor-cores 1 --executor-memory 1G
  6. --driver-java-options "-Dlog4j.configuration=file:/home/hadoop/spark-workspace/log4j-driver.properties "
  7. --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/spark-workspace/log4j-executor.properties"
  8. /home/hadoop/spark-workspace/my-spark-etl-assembly-1.0-SNAPSHOT.jar

注意：如果使用文件， file: 则应明确提供配置文件的，并且文件需要在所有节点上本地存在。