一、nodemanager异常日志

    1. 2020-08-11 15:20:41,563 WARN  ipc.Client (Client.java:run(752)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for dr_yangl: HDFS_DELEGATION_TOKEN owner=dr_yangl, renewer=ocdp, realUser=hive/oc-yx-hdp-19-55@ynmobile.com, issueDate=1597111276893, maxDate=1597716076893, sequenceNumber=71093278, masterKeyId=648) can't be found in cache
    2. 2020-08-11 15:20:41,564 ERROR filecontroller.LogAggregationFileController (LogAggregationFileController.java:run(361)) - Failed to setup application log directory for application_1597070923507_5403
    3. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for dr_yangl: HDFS_DELEGATION_TOKEN owner=dr_yangl, renewer=ocdp, realUser=hive/oc-yx-hdp-19-55@ynmobile.com, issueDate=1597111276893, maxDate=1597716076893, sequenceNumber=71093278, masterKeyId=648) can't be found in cache
    4.         at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
    5.         at org.apache.hadoop.ipc.Client.call(Client.java:1443)
    6.         at org.apache.hadoop.ipc.Client.call(Client.java:1353)
    7.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
    8.         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    9.         at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source)
    1. 2020-08-11 15:23:49,911 WARN  webapp.GenericExceptionHandler (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
    2. javax.ws.rs.WebApplicationException: org.apache.hadoop.yarn.webapp.NotFoundException: java.lang.Exception: Application: application_1597070923507_9907 is not found
    3.         at org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:185)
    4.         at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
    5.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    6.         at java.lang.reflect.Method.invoke(Method.java:498)
    7.         ...
    8.         Caused by: org.apache.hadoop.yarn.webapp.NotFoundException: java.lang.Exception: Application: application_1597070923507_9907 is not found
    9.         at org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:171)
    10.         ... 52 more
    11. Caused by: java.lang.Exception: Application: application_1597070923507_9907 is not found
    12.         at org.apache.hadoop.yarn.webapp.NotFoundException.<init>(NotFoundException.java:45)
    13.         ... 53 more
    14. 2020-08-11 15:23:49,963 ERROR impl.TimelineV2ClientImpl (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline server is not successful, HTTP error code: 403, Server response:
    15. {"exception":"ForbiddenException","message":"java.lang.Exception: The owner of the posted timeline entities is not set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
    16. 2020-08-11 15:23:50,025 ERROR impl.TimelineV2ClientImpl (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline server is not successful, HTTP error code: 403, Server response:

    二、 ResourceManager异常日志

    1. 2020-08-11 16:38:29,174 ERROR impl.TimelineWriter (TimelineWriter.java:doPosting(133)) - Failed to get the response from the timeline server.
    2. 2020-08-11 16:38:29,174 ERROR metrics.TimelineServiceV1Publisher (TimelineServiceV1Publisher.java:putEntity(372)) - Error when publishing entity [YARN_APPLICATION,application_1597070923507_11231]
    3. org.apache.hadoop.yarn.exceptions.YarnException: Failed to get the response from the timeline server. HTTP error code: 403
    4.         at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:142)
    5.         at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.putEntities(TimelineWriter.java:92)
    6.         at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:177)
    7.         at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.putEntity(TimelineServiceV1Publisher.java:370)
    8.         at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher.access$100(TimelineServiceV1Publisher.java:52)
    9.         at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher$TimelineV1EventHandler.handle(TimelineServiceV1Publisher.java:395)
    10.         at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV1Publisher$TimelineV1EventHandler.handle(TimelineServiceV1Publisher.java:391)
    11.         at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
    12.         at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
    13.         at java.lang.Thread.run(Thread.java:748)
    14. 2020-08-11 16:38:29,175 INFO  capacity.CapacityScheduler (CapacityScheduler.java:tryCommit(2906)) - Failed to accept allocation proposal

    三、timeline service异常日志

    1. 2020-08-10 22:49:20,265 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x7f2cfe3f no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    2. 2020-08-10 22:49:20,268 INFO  zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x36f658570feee21 closed
    3. 2020-08-10 22:49:20,268 INFO  zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
    4. 2020-08-10 22:49:20,272 INFO  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor
    5. 2020-08-10 22:49:20,485 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x73e22a3d no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    6. 2020-08-10 22:49:20,487 INFO  zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x36f658570feee20 closed
    7. 2020-08-10 22:49:20,487 INFO  zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
    8. 2020-08-10 22:50:20,269 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x7f2cfe3f no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    9. 2020-08-10 22:50:20,272 INFO  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor
    10. 2020-08-10 22:50:20,487 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x73e22a3d no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    11. 2020-08-10 22:51:20,270 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x7f2cfe3f no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    12. 2020-08-10 22:51:20,272 INFO  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor
    13. 2020-08-10 22:51:20,488 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x73e22a3d no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.
    14. 2020-08-10 22:52:20,271 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x7f2cfe3f no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests.

    四、zookeeper异常日志

    1. 2020-08-11 16:12:55,722 - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16f658570fff67c with negotiated timeout 60000 for client /10.170.21.142:33186
    2. 2020-08-11 16:12:55,829 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@120] - Successfully authenticated client: authenticationID=itd_model@ynmobile.com;  authorizationID=itd_model@ynmobile.com.
    3. 2020-08-11 16:12:55,829 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: itd_model
    4. 2020-08-11 16:12:55,829 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1038] - adding SASL authorization for authorizationID: itd_model
    5. 2020-08-11 16:12:55,864 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception
    6. EndOfStreamException: Unable to read additional data from client sessionid 0x16f658570fff67c, likely client has closed socket
    7.         at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
    8.         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
    9.         at java.lang.Thread.run(Thread.java:748)
    10. 2020-08-11 16:12:55,864 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /10.170.21.142:33186 which had sessionid 0x16f658570fff67c